[#10029][benchmarks] arrow-flight roundtrip as well as encode/decode

[Rich-T-kid]

Which issue does this PR close?

• Contributes towards closing Optimize arrow-ipc #10029.

Rationale for this change

Provides benchmarks for arrow-flight crate. benchmarks for round trip as well as encode/decode individually.

What changes are included in this PR?

Adds three criterion benches under arrow-flight/benchmarks/ (roundtrip.rs, flight_encode.rs, flight_decode.rs), each sweeping a tunable matrix of rows, cols, and column types (fixed Int64, variable StringArray, nested List, dict DictionaryArray) built via a shared
common::build_batch helper.

Are these changes tested?

n/a

Are there any user-facing changes?

no

[Rich-T-kid]

@alamb could you take a look at this when you get a chance? smaller PR that only includes benchmarks, #10029 has a bit more context. 🫡

[gabotechs]

👍 nice work @Rich-T-kid! pretty clean benchmarks. Left some suggestions, but otherwise this LGTM

[alamb]

Thanks @Rich-T-kid and @gabotechs

I think this looks good to me after we implement @gabotechs 's suggestion

[Rich-T-kid]

pushed a revised PR @alamb. im not sure why the CI is failing. I'm running ./regen.sh but that isnt resolving it.

[alamb]

pushed a revised PR @alamb. im not sure why the CI is failing. I'm running ./regen.sh but that isnt resolving it.

Looks like the fail is https://github.com/apache/arrow-rs/actions/runs/26834183268/job/79123485734?pr=10031

Run git diff --exit-code
diff --git a/Cargo.lock b/Cargo.lock
index 28fe6a4..9e5963a 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -348,6 +348,7 @@ dependencies = [
  "base64",
  "bytes",
  "clap",
+ "criterion",
  "futures",
  "http",
  "http-body",
@@ -1063,6 +1064,7 @@ dependencies = [
  "serde",
  "serde_json",
  "tinytemplate",
+ "tokio",
  "walkdir",
 ]

do we need to check in the Cargo.lock file 🤔

[Rich-T-kid]

yea the error mentions the cargo.lock file but it doesn't exist within the arrow-flight repo.

[Rich-T-kid]

The issue is that the cargo.lock file on main doesnt have criterion listed as a dependency for arrow-flight. But Cargo.lock is ignored due to the .gitignore so I cant push an update directly. ./arrow-flight/regen.sh is suppose to do this but it only shows up in my local environment.

[Rich-T-kid]

😰 fixed it

[alamb]

Thanks @Rich-T-kid -- I started going through this one.

I suggest:

1. Put the code in single benchmark for now arrow-flight/benches/flight.rs
2. Name the benchmarks after whatever flight command they are doing (put and exchange for example)
3. Run the benchmarks locally (via cargo bench --bench <name> ) and profile them to make sure they show that the encoder is taking substantial time

[alamb]

this seems to be benchmarking do_put not really flight_encode 🤔

[Rich-T-kid]

Originally I wanted to do an end-to-end benchmark, but I think it makes more sense to have smaller, more focused benchmarks. I updated the benchmarks to include one round-trip benchmark and another that benchmarks FlightDataEncoder::encode_batch() directly. Other benchmarks can be added later organically.

[Rich-T-kid]

majority of the time is being spent int arrow-ipc as expected, this is what #10044 aims to address. its not directly wired up to to arrow-flight yet, but the results seems to be good so far.

[Rich-T-kid]

CI is finally passing, sorry about that! @alamb should be good now, lmk if theres anything else

[alamb]

Thank you @Rich-T-kid -- while it did take some back and forth I think this benchmark will now let us improve things substnatially.

I agree with your assessment that there is a lot of copying going on -- it will be great to see PRs to avoid it

[alamb]

Thank you aslo @gabotechs for your review