AWB with Neural Networks by davidplowman · Pull Request #292 · raspberrypi/libcamera

davidplowman · 2025-08-04T07:34:58Z

Initial implementation of AWB using neural networks. This is an alternative to the old Bayesian method.

davidplowman · 2025-08-04T07:37:19Z

src/ipa/rpi/controller/rpi/awb.cpp

+		greyWorld = false;
+	} else {
+		greyWorld = true;
 		sensitivityR = sensitivityB = 1.0; /* nor do sensitivities make any sense */


What's the story with neural network AWB and sensitivities, are we stll respecting them?

davidplowman · 2025-08-04T07:42:35Z

src/ipa/rpi/controller/rpi/awb.cpp

- * elsewhere (ALSC and AGC).
- */
-
-int AwbMode::read(const libcamera::YamlObject &params)


Maybe we should put this method back here in the cpp file?

davidplowman · 2025-08-04T07:48:24Z

src/ipa/rpi/controller/rpi/awb.cpp

 	for (auto &z : zones_) {
-		double deltaR = gainR * z.R - 1 - config_.whitepointR;
-		double deltaB = gainB * z.B - 1 - config_.whitepointB;
+		double deltaR = gainR * z.R - whitepointR;


Just wondering what happened to the -1 here... is that included in the whitepointR/B values now?

davidplowman · 2025-08-04T07:51:12Z

src/ipa/rpi/controller/rpi/awb.h

 struct AwbMode {
-	int read(const libcamera::YamlObject &params);
+	int read(const libcamera::YamlObject &params)
+	{


Maybe move back to cpp?

davidplowman · 2025-08-04T07:59:10Z

src/ipa/rpi/controller/rpi/awb.cpp

-	 * Given 3 points on a curve, find the extremum of the function in that
-	 * interval by fitting a quadratic.
-	 */
+	* Given 3 points on a curve, find the extremum of the function in that


Not 100% convinced by the indentation here...

davidplowman · 2025-08-04T08:06:25Z

src/ipa/rpi/controller/rpi/awb.cpp

-	generateStats(zones_, statistics_, config_.minPixels,
-		      config_.minG, getGlobalMetadata(),
-		      config_.biasProportion, biasCtR, biasCtB);
+	generateStats(zones_, statistics_, 0.0, 0.0, getGlobalMetadata(), 0.0, 0.0, 0.0);


AwbBayes seems to have its own prepareStats, is that right? So is this the AwbNN version? Might be better to leave it pure virtual in the base class and define it in awb_nn.cpp.

davidplowman · 2025-08-04T08:08:52Z

src/ipa/rpi/controller/rpi/awb.cpp

 				continue;
 			zone.R = region.val.rSum / region.counted;
 			zone.B = region.val.bSum / region.counted;
 			/*


Has the indentation gone wrong here?

davidplowman · 2025-08-04T08:10:26Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+	float minTemp;
+	float maxTemp;
+
+	bool enable_nn;


libcamera will want enableNn instead of enable_nn.

davidplowman · 2025-08-04T08:14:04Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+			modelPath = root + modelPath;
+		} else {
+			modelPath = LIBCAMERA_DATA_DIR + modelPath;
+		}


Might be worth printing out the model path in an Info statement. People waste so much time thinking they're loading something that they aren't (myself included)!

Ah I see we do it once it's loaded. Well, myabe a Debug statement then, so that folks can double-check if it's not being found.

davidplowman · 2025-08-04T08:16:14Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+	}
+}
+
+void AwbNN::transverseSearch(double t, double &r, double &b)


I sometimes wonder if we should do a very short search along the curve too, but not sure. TBD, I guess.

davidplowman · 2025-08-04T08:17:22Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+
+static double getElementPadded(const std::vector<double> &array, int i, int j)
+{
+	int i_padded = std::clamp(i, 0, 31);


libcamera will want snakecase iPadded.

davidplowman · 2025-08-04T08:17:41Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+	 * Decreases the difference between the colour channels when it is higher
+	 * than nearby pixels.
+	 */
+	const int total_pixels = 32 * 32;


Same throughout this function, totalPixels etc.

davidplowman · 2025-08-04T08:19:26Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+
+void AwbNN::awbNN()
+{
+	float *input_data = interpreter_->typed_input_tensor<float>(0);


inputData etc.

davidplowman · 2025-08-04T08:22:12Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+			enable_nn = false;
+		}
+
+		bool valid_ccm = true;


validCcm etc.

davidplowman · 2025-08-04T08:23:21Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+		return;
+	}
+
+	const int expected_input_size = 32 * 32 * 3;


libcamera will want expectedInputSize. Same for all the others...

davidplowman · 2025-08-04T08:27:26Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+	zone_log.B += 1.0;
+
+	zone_log.R /= 2.0;
+	zone_log.B /= 2.0;


Is there any particular rationale for these adjustment? Might be worth a comment.

davidplowman · 2025-08-04T08:30:13Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+	for (int i = 0; i < num_outputs; i++) {
+		float temp = min_temp + i * (max_temp - min_temp) / (num_outputs - 1);
+		float weight = output_data[i];
+		weight *= weight;


Interesting. Makes all the weights +ve, among other things! I wonder a bit about putting a softmax on the end of the network instead, is that something we've tried?

Move parts of the AWB algorithm specific to the Bayesian algorithm into a new class. This will make it easier to add new Awb algorithms in the future. Signed-off-by: Peter Bailey <peter.bailey@raspberrypi.com>

Add an Awb algorithm which uses neural networks. Signed-off-by: Peter Bailey <peter.bailey@raspberrypi.com>

Update the tuning files to include the new Awb algorithm. It is enabled by renaming disable.rpi.nn.awb to rpi.nn.awb and rpi.awb to disable.rpi.awb. Add a model for the Awb algorithm to use by default. Signed-off-by: Peter Bailey <peter.bailey@raspberrypi.com>

Prevent an algorithm starting with "disable" from being loaded. Signed-off-by: Peter Bailey <peter.bailey@raspberrypi.com>

davidplowman · 2025-09-03T08:24:34Z

src/ipa/rpi/controller/rpi/awb.cpp

+	ctHi = *value;
+
+	return 0;
+}


Super-minor nitpick. If we put this above above readCtCurve would that minimise the diffs a bit?

davidplowman · 2025-09-03T08:27:41Z

src/ipa/rpi/controller/rpi/awb.cpp

+		greyWorld = false;
+	} else {
+		greyWorld = true;
 		sensitivityR = sensitivityB = 1.0; /* nor do sensitivities make any sense */


The CT curve belongs to the base AWB class, just want to check if we think the sensitivities should belong with the CT curve or not?

davidplowman · 2025-09-03T08:34:34Z

src/ipa/rpi/controller/rpi/awb_bayes.cpp

+		}
+	}
+	whitepointR = params["whitepoint_r"].get<double>(0.0);
+	whitepointB = params["whitepoint_b"].get<double>(0.0);


I guess I wonder a little bit if the whitepoint might not be a base class thing. What do we think?

davidplowman · 2025-09-03T08:40:57Z

src/ipa/rpi/controller/rpi/awb_nn.cpp

+
+		bool validCcm = true;
+		for (int i = 0; i < 9; i++)
+			if (ccm[i] == 0.0)


I think you can have zeroes in a CCM. But every row does have to sum to 1 (otherwise it doesn't preserve greyness).

davidplowman · 2025-09-03T11:02:50Z

src/ipa/rpi/pisp/data/imx219.json

+                    -0.2613746357615894, -0.668015927152318, 1.9293905629139072
+                ],
+                "enable_nn": 1
+            }


Might there be any other useful options? I wonder about a "run on one/all core(s)" option, or is it so quick it's not worth bothering?

davidplowman commented Aug 4, 2025

View reviewed changes

PineappleBeech force-pushed the awb-nn branch 3 times, most recently from e283aef to 0b3ea0b Compare August 20, 2025 13:23

ipa: rpi: controller: awb: Separate Bayesian Awb into AwbBayes

45798ff

Move parts of the AWB algorithm specific to the Bayesian algorithm into a new class. This will make it easier to add new Awb algorithms in the future. Signed-off-by: Peter Bailey <peter.bailey@raspberrypi.com>

PineappleBeech force-pushed the awb-nn branch 2 times, most recently from edabec9 to 202185e Compare August 29, 2025 09:08

PineappleBeech added 3 commits September 1, 2025 11:20

ipa: rpi: controller: awb: Add Neural Network Awb

7e16bd3

Add an Awb algorithm which uses neural networks. Signed-off-by: Peter Bailey <peter.bailey@raspberrypi.com>

ipa: rpi: controller: Ignore algorithms starting with disable

a68be7e

Prevent an algorithm starting with "disable" from being loaded. Signed-off-by: Peter Bailey <peter.bailey@raspberrypi.com>

PineappleBeech force-pushed the awb-nn branch from 202185e to a68be7e Compare September 1, 2025 16:01

davidplowman commented Sep 3, 2025

View reviewed changes

davidplowman closed this Feb 5, 2026

Conversation

davidplowman commented Aug 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants