Uploaded file is broken, file upload fails or uploaded file are gone with OneDrive resumable API to upload large files by chunks

[denven]

Seems it happened today:

1. the first-time, the video file is uploaded with full size, but cannot be played
   

2. curl throws an error of the ongoing uploads and no files can actually be uploaded (Previously, it seems this was just a warning message but the file was still uploaded)

curl: option --data-binary: out of memory
curl: try 'curl --help' for more information
upload chunk: 1, length: 5898240, bytes: 0-5898239
curl: option --data-binary: out of memory
curl: try 'curl --help' for more information
upload chunk: 2, length: 3550165, bytes: 5898240-9448404

• the full curl information is below:

Start to upload /tmp/sd/record/2023Y01M14D10H/06M24S36.mp4
Success: create_upload_session /tmp/sd/record/2023Y01M14D10H/06M24S36.mp4
upload_large_file_by_chunks /tmp/sd/record/2023Y01M14D10H/06M24S36.mp4, 6.29MB
*   Trying 13.107.42.12:443...
* TCP_NODELAY set
* Connected to api.onedrive.com (13.107.42.12) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* error setting certificate verify locations, continuing anyway:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; OU=Microsoft Corporation; CN=storage.live.com
*  start date: Jun 13 20:46:06 2022 GMT
*  expire date: Jun  8 20:46:06 2023 GMT
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x263d30)
> PUT /rup/be8ddd0b7f4904f6/eyJSZXNvdXJjZUlEIjoiQkU4REREMEI3RjQ5MDRGNiE0MjA2IiwiUmVsYXRpb25zaGlwTmFtZSI6IjA2TTI0UzM2Lm1wNCJ9/4macv3OHZ2mMNWkT0ZP1VQrIS8At8uAONk5CPDtApUn7aq5antryO-PE6gXJQzokuGl36s79ac55F-o_GqlMErptYaXSBkdZsE5hr-JAl7QWg/eyJuYW1lIjoiMDZNMjRTMzYubXA0IiwiQG5hbWUuY29uZmxpY3RCZWhhdmlvciI6InJlcGxhY2UifQ/4wqRpwZTj3-HeGRk59wK16jYtmt-vrvXFgbUOq33ro4h8G2qAFfOd1O_p1gg-ANuQaeEtfy5Bs0Gs2L8RGQhY31aw4zjZd5fmsOzhAX3YXzxYe4gfQr_U-9qfVg-8jcw27Ihxw5W2wzXFVUDF8AMHubYXqdDydQhzG1zWNpejvByJ_x7m9eoD040qPdKy379X-X8pcD13S6hsMiVeEWRCHJl_bEN4UgDl78VgdCUTiye_0TiC01R6O7jyTkP_HXoPbDpDBM0vozoRD3AJ4aYZPfg1t2jixgltW7P-_WWlwF8I73OzqLmLCpEvNaCEMB7yJWuO8qOul6jjZo2VR09EXkgyKPH8lEIOBy94ehbWb3k0SLgr7rPO4aZCYph9CZSupwUasuzxTZZSkXCY_-W1gKwsG2hWVXqaHieUmTVF4j7d1mNgzE3bxFq-5GxQYVy9anh9b3XzjddTjEEKzp9AhE8xAfEI-uP_EyyqGv4noMFPGVVSl9mBz7BnwSU-R1iWW8jniWuiFbJ1Njd0zpzg2rfLzKyEb6Fs37xo5bJWxx7qFcfYatNZVifj3VNnwnZupccdIuJTi5aEyCFA0BJlVNw HTTP/2
> Host: api.onedrive.com
> user-agent: curl/7.68.0
> accept: */*
> content-length: 5898240
> content-range: bytes 0-5898239/6595575
> content-type: application/x-www-form-urlencoded
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 202
< content-length: 90
< content-type: application/json; charset=utf-8
< p3p: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
< x-msnserver: BN6PPF90571263D
< strict-transport-security: max-age=31536000; includeSubDomains
< ms-cv: hq/Jwns3jUSX46FTqzSHGQ.0
< x-asmversion: UNKNOWN; 19.1058.1201.2010
< x-cache: CONFIG_NOCACHE
< x-msedge-ref: Ref A: 7D9FFEC337AA44B6B0F94B69F081252A Ref B: YVR311000109027 Ref C: 2023-01-15T01:50:08Z
< date: Sun, 15 Jan 2023 01:50:21 GMT
<
* We are completely uploaded and fine
* Connection #0 to host api.onedrive.com left intact
{"expirationDateTime":"2023-01-20T01:50:07.271Z","nextExpectedRanges":["5898240-6595574"]}upload chunk: 1, length: 5898240, bytes: 0-5898239
*   Trying 13.107.42.12:443...
* TCP_NODELAY set
* Connected to api.onedrive.com (13.107.42.12) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* error setting certificate verify locations, continuing anyway:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; OU=Microsoft Corporation; CN=storage.live.com
*  start date: Jun 13 20:46:06 2022 GMT
*  expire date: Jun  8 20:46:06 2023 GMT
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x263d40)
> PUT /rup/be8ddd0b7f4904f6/eyJSZXNvdXJjZUlEIjoiQkU4REREMEI3RjQ5MDRGNiE0MjA2IiwiUmVsYXRpb25zaGlwTmFtZSI6IjA2TTI0UzM2Lm1wNCJ9/4macv3OHZ2mMNWkT0ZP1VQrIS8At8uAONk5CPDtApUn7aq5antryO-PE6gXJQzokuGl36s79ac55F-o_GqlMErptYaXSBkdZsE5hr-JAl7QWg/eyJuYW1lIjoiMDZNMjRTMzYubXA0IiwiQG5hbWUuY29uZmxpY3RCZWhhdmlvciI6InJlcGxhY2UifQ/4wqRpwZTj3-HeGRk59wK16jYtmt-vrvXF1gg-ANuQaeEtfy5Bs0Gs2L8RGQhY31aw4zjZd5fmsOzhAX3YXzxYe4gfQr_U-9qfVg-8jcw27Ihxw5W2wzXFVUDF8AMHubYXqdDydQhzG1zWNpejvByJ_x7m9eoD040qPdKy379X-X8pcD13S6hsMiVeEWRCHJl_bEN4UgDl78VgdCUTiye_0TiC01R6O7jyTkP_HXoPbDpDBM0vozoRD3AJ4aYZPfg1t2jixgltW7P-_WWlwF8I73OzqLmLCpEvNaCEMB7yJWuO8qOul6jjZo2VR09EXkgyKPH8lEIOBy94ehbWb3k0SLgr7rPO4aZCYph9CZSupwUasuzxTZZSkXCY_-W1gKwsG2hWVXqaHieUmTVF4j7d1mNgzE3bxFq-5GxQYVy9anh9b3XzjddTjEEKzp9AhE8xAfEI-uP_EyyqGv4noMFPGVVSl9mBz7BnwSU-R1iWW8jniWuiFbJ1Njd0zpzg2rfLzKyEb6Fs37xo5bJWxx7qFcfYatNZVifj3VNnwnZupccdIuJTi5aEyCFA0BJlVNw HTTP/2
> Host: api.onedrive.com
> user-agent: curl/7.68.0
> accept: */*
> content-length: 697335
> content-range: bytes 5898240-6595574/6595575
> content-type: application/x-www-form-urlencoded
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 201
< content-length: 2273
< content-type: application/json; charset=utf-8
< p3p: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
< x-msnserver: BN6PPF11E39FAF5
< strict-transport-security: max-age=31536000; includeSubDomains
< ms-cv: JVEanuotCUawcZsRhcg0kQ.0
< x-asmversion: UNKNOWN; 19.1058.1201.2010
< x-cache: CONFIG_NOCACHE
< x-msedge-ref: Ref A: 4455123FA14941A48CB62A84C01078B8 Ref B: YVR311000112019 Ref C: 2023-01-15T01:50:22Z
< date: Sun, 15 Jan 2023 01:50:23 GMT
<
{"createdBy":{"application":{"id":"44d45bee"},"user":{"id":"be8ddd0b7f4904f6"}},"createdDateTime":"2023-01-15T01:50:23.84Z","cTag":"aYzpCRThEREQwQjdGNDkwNEY2ITQyMDguMjU3","eTag":"aQkU4REREMEI3RjQ5MDRGNiE0MjA4LjA","id":"BE8DDD0B7F4904F6!4208","lastModifiedBy":{"application":{"id":"44d45bee"},"user":{"id":"be8ddd0b7f4904f6"}},"lastModifiedDateTime":"2023-01-15T01:50:23.84Z","name":"06M24S36.mp4","parentReference":{"driveId":"be8ddd0b7f4904f6","driveType":"personal","id":"BE8DDD0B7F4904F6!4206","name":"2023Y01M14D10H","path":"/drive/root:/yihack_videos/2023Y01M14D10H"},"size":6595575,"webUrl":"https://1drv.ms/i/s!APYESX8L3Y2-oHA","items":[],"file":{"hashes":{"quickXorHash":"kKSjmO45QijFJDhODrM/ZO5EMqU=","sha1Hash":"D71DE76A73C6A374A20FE4B44E89F1DDEEF25573","sha256Hash":"7A5F976074A06CDBB00786BB2EF1B069142A191E1DAEA6A9D93CD070727DD157"},"mimeType":"video/mp4","processingMetadata":true},"fileSystemInfo":{"createdDateTime":"2023-01-15T01:50:23.84Z","lastModifiedDateTime":"2023-01-15T01:50:23.84Z"},"reactions":{"commentCount":0},"photo":{},"tags":[],"lenses":[],"thumbnails":[{"id":"0","large":{"height":800,"url":"https://yvkzbg.bn.files.1drv.com/y4pR8cf35cf-VLa7ZZurIc2Y7rVNYl6k0u-1aKqZftxI3j9VuPVhlTMRZYrdtYb55a3OpL7wyBUV13kwi11EonXYH9lZuYA81n8-i3FoZkOx5VUYBUcUoZY7jmBv3Z_S5mRjug0JEE9FB-tYuXlJTyrhkCF4M0vzu9Yyuz0MUrahzKEyNkSA4S1LIKzn4aUwHdqU3LKG8PUXEhOG-68XAs9aP-h30kdCmm6vsEd5lXf5rbGC8PZ1nPy_hsqzjF1x-lIhXSv1oeEG6MhB8L3GV8b5A?width=800&height=800&cropmode=none","width":800},"medium":{"height":176,"url":"https://yvkzbg.bn.files.1drv.com/y4pR8cf35cf-VLa7ZZurIc2Y7rVNYl6k0u-1aKqZftxI3j9VuPVhlTMRZYrdtYb55a3OpL7wyBUV13kwi11EonXYH9lZuYA81n8-i3FoZkOx5VUYBUcUoZY7jmBv3Z_S5mRjug0JEE9FB-tYuXlJTyrhkCF4M0vzu9Yyuz0MUrahzKEyNkSA4S1LIKzn4aUwHdqU3LKG8PUXEhOG-68XAs9aP-h30kdCmm6vsEd5lXf5rbGC8PZ1nPy_hsqzjF1x-lIhXSv1oeEG6MhB8L3GV8b5A?width=176&height=176&cropmode=none","width":176},"small":{"height":96,"url":"https://yvkzbg.bn.files.1drv.com/y4pR8cf35cf-VLa7ZZurIc2Y7rVNYl6k0u-1aKqZftxI3j9VuPVhlTMRZYrdtYb55a3OpL7wyBUV13kwi11EonXYH9lZuYA81n8-i3FoZkOx5VUYBUcUoZY7jmBv3Z_S5mRjug0JEE9FB-tYuXlJTyrhkCF4M0vzu9Yyuz0MUrahzKEyNkSA4S1LIKzn4aUwHdqU3LKG8PUXEhOG-68XAs9aP-h30kdCmm6vsEd5lXf5rbGC8PZ1nPy_hsqzjF1x-lIhXSv1oeEG6MhB8L3GV8b5A?width=96&height=96&cropmode=none","width":96}}]}* We are completely uploaded and fine
* Connection #0 to host api.onedrive.com left intact
upload chunk: 2, length: 697335, bytes: 5898240-6595574
Success: upload_large_file_by_chunks /tmp/sd/record/2023Y01M14D10H/06M24S36.mp4
Start to upload /tmp/sd/record/2023Y01M14D10H/07M00S60.mp4
Success: create_upload_session /tmp/sd/record/2023Y01M14D10H/07M00S60.mp4
upload_large_file_by_chunks /tmp/sd/record/2023Y01M14D10H/07M00S60.mp4, 10.29MB
curl: option --data-binary: out of memory
curl: try 'curl --help' for more information
upload chunk: 1, length: 5898240, bytes: 0-5898239
curl: option --data-binary: out of memory
curl: try 'curl --help' for more information
upload chunk: 2, length: 4886500, bytes: 5898240-10784739

[denven]

for files with a size greater than 4Mb, I changed to send the file in one request without split by chunks, there is one file uploaded that can be played, but all others were all not uploaded to OneDrive and throw the same error message out of memory.

[denven]

The camera has very limited memory resources, then when curl sends large files, the available memory for applications like curl to use is less, then it is easy for curl runs into an out of memory issue. That makes sense.

However,

• Even if I set the chunk size as 320KB, it still throws the error message, so it doesn't make sense;

• When I call the dd command to pass the same chunk size of data to curl, it doesn't give any errors; (anyway I won't use this way if it is not the last straw, because it will drain out of the CPU resource.
  

• And I also got one 6.28MB file uploaded successfully without giving any errors, and the file is split into 2 chunks, the first chunk is 5.625MB.

I guess it is kind of a protection mechanism of curl which will be activated under some circumstances, but it is not done well. Even though curl can support sending files as large as GB size, when it meets the resource-constraint running env, it is not acclimatized.

[denven]

Why the uploaded data is gone? because OneDrive API will drop data it considers to be not in the right order.

FragmentOutOfOrder

{
    "error": {
        "code": "invalidRange",
        "message": "The uploaded fragment is not contiguous with the last one.",
        "innererror": {
            "code": "fragmentOutOfOrder"
        }
    }
}

This is a little ridiculous since I send the data in sequence and even when I have 2 chunks sent in sequence with a delay time between, it also has a chance to throw this error.

However, when each time the file is uploaded in 1 or 2 chunks only, the chance of error is fewer.

[denven]

Solution:

Since the OneDrive resumable API cannot assure the file chunks are in the right order, even when we send them in sequence.

1. for this case, I added the retransmission of chunks when finding a "FragmentOutOfOrder" with max 5 re-tries. This will avoid the chance of failure of merging the video file chunks by OneDrive or broken files.

in order to get rid of the `out of memory error on the resource constraint camera, I find a moderate way after considering some tradeoffs.

2. since the resumable API uses the PUT method, and the maximum data in a single push can be 64MB, but when the size of reading data is greater than 10MB, the/my camera has a chance of reboot, so we cannot read one whole file and send it once to save time.

3. Using dd command to read part of the video file into curl is simple, but it consumes more CPU resources, the combination use of the two commands will make the CPU utilized rate stay constantly more than 90%, and the camera is very likely to run into frequent reboots. In order to read the video file in chunks, I use a temporary file to save the chunk data copied from dd command, and let curl read the chunk data each time. In this way, the CPU spike can decline by 10-20% (although it has short time running more than 90%) because the 2 commands run separately. However, this will increase the SD card IO operations.
   

4. for the chunk size set it as 3276800 bytes, which means one chunk is 3.125MB, and one video file may need 4 times of data push to upload.

After doing the above 4 measures:

• The memory usage is in control, no more `out of memory errors
• The CPU spikes are around 90%, so it is less likely to get a camera reboot
• The upload speed is not slow, but not fast as well, which can be accepted
• Video files upload will seldom fail

Finally, the issue is resolved. 👏