backend: Add Apple iCloud Drive backend #7717
Conversation
|
Nice work :-) Some quick questions
Thank you |
not sure what client_id you are referring to? and no I havent run any integration test yet. |
|
fyi: so iCloud doesn't have functionality for updating/replacing files. At least not something I can find reverse engineering So if a file exists I move it to iCloud trash. And upload a new version of the file. |
|
@ncw how one handle folder invalidation? drive uses etags for files and folders. If a file or folder within a folder changes then the parent folders etag changes. This means I have to clear almost the whole tree from child to parent. Not sure how to handle that in general, what if a users change it locally but file/folder has changed remotely in between. What error should I raise on that? (if etags are not matching) cachedir doesn't seem to refresh after invalidation, see: rclone/backend/icloud/icloud.go Line 424 in 37816e9 When calling flushdir at the end of rmdir I see the call for FindLeaf but the cachedir entries aren't updated. |
This one at the top of the source! clientId = "e9f98057fb916de2bbd755ef280d7257146a76e5118f27ab2e9a3d065c20c17e"
Running the integration tests will guide you through what you need to do. You can start running them as soon as you've done
Seems OK!
Do you have to use the etags? They are optional for most cloud storage systems and rclone doesn't use them. This means that the last write wins which is what we want. If you can't ignore etags then refresh the etag and try again.
Not sure! |
etags are needed for file operations unfortunately, but I can just refresh it when it got stale. the clientid is a static that icloud.com uses. So I reuse it to mimic the browser. |
|
@ncw small question. iCloud Drive does not allow uploading empty files so check the size and raise ErrorCantUploadEmptyFiles when its 0 But when using macOS finder to copy a file, I get an partial file with zero length (while the actually file isn't empty) Not sure howto handle this. |
That sounds correct.
Are you using So you are seeing a zero length file in the finder? I'm not sure what you mean by a partial file? Can you reproduce this without the |
|
Its slow but finished alle interfaces and all integration test are passing 🎉 Only documentation left before its ready for review. |
Well done! You are running the full suite from
:-) |
|
@lostb1t not sure if it's too early for bug reports, but currently it looks like app folders aren't being represented as folders, but rather as files. |
bug reports are welcome. Will fix with the following pass, tnx |
|
@epetousis this has been fixed |
|
crap i broke my icloud. Too many trash items and now the endpoint is broken lol the icloud web api is a mess |
|
Fix works great. Much appreciated! |
|
Hi, thanks. |
@cyayon Could you post a log from the command with the -vv flag (e.g. output from rclone -vv ls |
|
maybe removing and readding ylur remote might work (new credentials) if not then im blaming Apple lol, try again tomorrow? |
|
I wil try again tomorrow. it is different from yours. |
|
just tried to replace clienId in backend/icloud/api/client.go with mine clientId, and test. no change, same issue. |
|
I don't know if this could help about clientId : https://backstage.forgerock.com/docs/sdks/latest/sdks/use-cases/social-login/creatingappleclient.html https://developer.apple.com/documentation/mediasetup/msserviceaccount/clientid# |
|
Since this morning I am seeing a very similar error. |
|
ClientId ban by apple ? |
|
Hi, |
|
does downloading the file from icloud.com work? |
|
Yes of course |
|
well its the same api rclone uses so if that works then might be on my side. Might be they are blocking yeah, figure out on what, might be the user agent. could you try with something like
|
|
Hi, |
|
Hi, I tried to fully re-create config and re-authenticate on iCloud, also with my "own" clientId (found in headers on iCloud.com). |
I'm getting the same errors here. |
|
Alright this should be fixed. At least in my testing, let me know if it still occures. PS the old way def worked before as i run all the unit test without issue. And where now failing. So either Apple decided to kill it or they borked the endpoint. Either way, now using (a slower) other method |
|
Hi, With the last patch today, it is better. Some download work as expected. |
|
after the last commit 98ebc42, I got the same error. when I relaunch (same directory source) : |
|
@cyayon thanks for pointing me in the right direction, this was a bug with application files and has been fixed. |
|
Great, perfect ! Just a suggestion, I think it would be a good thing if we you could specify a different clientId in config (rclone config question). My clientId in web headers is different from yours (d39ba9916b725105xxxxxxxxxxxxxxxxd9d1a815d). |
|
yeah ill check it out and replace it when i have the change |
|
Great ! Another thing, I found the download very slow (about 1MB/s) for a large 3GB file. When using Finder, the download speed is much higher (about 20MB/s). Is there some reason ? Thank you very much. |
if you say finder so you mean apples native drive integration? |
|
Yes on macOS (and files.app on iOS). |
|
different apis. Apple probaply using a private api we dont have access to. This uses the web api. So if you want to compare use icloud.com, those speeds should |
|
I understand. |
|
With the current HEAD commit, I did When I try to copy a non-zero-bytes file into it, when mounted, it says ENV: Xubuntu 24.04 (6.8.0-31-generic) |
|
@hurryman2212 > do you maybe have advanced data protection enabled? thats currently not supported and need to be disabled. |
It has been almost a year since they introduced it and I enabled it, I think, so I have completely forgotten about it.. Thank you! Indeed, it was due to ADP. Now, it is working fine, although performance-wise, there are some issues as @cyayon reported. On my environment, |
What is the purpose of this change?
Add support for a new backend, iCloud Drive
Was the change discussed in an issue or in the forum before?
#1778
Checklist
Notes
This feature branch is in a working state, feedback on finished implementations is appreciated. Ill keep it in draft untill everything is implemented and ill update the TODO accordingly
This PR is only for iCloud Drive. Ill continue work on Photos once this PR is merged.
Todo
Rclone Interfaces
Fs
Object
Mover
DirMove
Copier