Rendered at 05:05:14 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
nomilk 17 hours ago [-]
> Three things to notice, because each one bites later:
Very LLMish language!
405error 17 hours ago [-]
It's that mix of dense, impressive sounding jargon, but even scanning across it raises glaring problems. Like, if you have 4.5 billion videos on HF, and it's 289GB, it's about 60 bytes per video. Checking the column fields as well, there doesn't seem to be any video files*.
nomilk 17 hours ago [-]
There's an `is_video` column, perhaps containing a lot of 0's
405error 16 hours ago [-]
More directly, there simply aren't any video files uploaded. It's just parquet files, which contain no video columns (I'm not even sure if it supports it).
nomilk 16 hours ago [-]
Seems the expectation is the purchaser uses the info/parameters in the 289gb file to decide which videos to download, then uses the /v1/video/info endpoint to download individual videos.
Retr0id 17 hours ago [-]
Can any brave soul wade through the LLM prose to provide a human-readable summary?
simonw 16 hours ago [-]
"I uploaded 4.5 billion of those videos to Hugging Face" = I scraped the metadata (title, view count, etc) for 4.5 billion TikTok videos using the same API as their Android app. I posted the data to HF as Parquet. The video content itself is not included. I'll sell you my Go scraping code.
405error 17 hours ago [-]
I cannot verify whether it is technically correct, but it's about how to defeat Tiktok's bot filters to scrape it.
arcfour 17 hours ago [-]
Man, I could use $699. If this gets even a single sale, then maybe I need to try having less decency...
(I guess that is sort of a roundabout summary...)
echoangle 17 hours ago [-]
Would you say that this is immoral? Since it’s data that’s public anyways, I don’t see why putting it in a table and selling it is a bad thing.
kenniskrag 15 hours ago [-]
Because a lot of things is public online and can't be copied and sold e.g. due to copyright, patents and trademark. Also if you access a website you are bound to a ToS contract and this is a breach of that contract.
echoangle 15 hours ago [-]
Breaching a contract isn’t a crime though, it’s a civil matter.
AlecSchueler 11 hours ago [-]
You asked if it was immoral, not if it was a crime.
breezybottom 17 hours ago [-]
What is "public" data? It admits to violating the ToS.
arcfour 17 hours ago [-]
Something posted with the understanding that it could potentially be viewed by anyone on the Internet with minimal/no restriction.
ToS are just what you follow if you don't want to get banned off of the site. If you don't care about that, then you can go hog wild, though you're being a bit of a jerk/not playing nice obviously.
breezybottom 17 hours ago [-]
Any company could potentially be hacked, so by that standard all data is public. And no, the ToS is legally binding on the company as well.
echoangle 15 hours ago [-]
You don’t see a difference between accessing data by hacking a service (which is illegal) and downloading/scraping data that’s not protected? The TOS are a contract and only a civil thing afaik.
arcfour 17 hours ago [-]
That's a blatant strawman - that's not a reasonable position at all. A reasonable person does not expect their private medical records to be accessible to you or me just because they are stored in an EHR system.
They might be surprised that you or I looked at their TikTok video when we aren't the intended audience, but they still posted it publicly, with the understanding that it would be made freely available to others.
breezybottom 16 hours ago [-]
It's definitely not a strawman. Perhaps you meant that it's a false equivalency, but I don't think that's true either.
With how common data hacks are, why wouldn't a reasonable person expect their medical records to leak? I received at least two such breach notices just last year.
arcfour 16 hours ago [-]
Obviously I meant that any data a person expects to be public, like a public post on a social media site... I was not referring to data exposed in data breaches, which is a different subject entirely...
If that was not obvious to you then I apologize; though it really should have been, since you are encouraged to interact with others in good faith on HN.
And if you expect your medical records to be public...then what is the point of this discussion?
breezybottom 16 hours ago [-]
The user only gave TikTok permission to use the videos according to the ToS. And considering the majority of TikTok users are children, I think it's hard to justify morally, even if you could make a legal case.
arcfour 16 hours ago [-]
A "data breach" refers to unauthorized access to nonpublic or protected data. Scraping content that is publicly viewable without logging in - or even with logging in, since an account is effectively disposable - is not a "data breach" (as far as any typical usage of the term goes).
In hiQ Labs v. LinkedIn, the 9th Circuit (US) ruled that scraping publicly accessible data does not violate the CFAA's "without authorization" clause (hiQ was bulk scraping public LinkedIn profile data - in violation of LinkedIn's ToS). The Supreme Court later specifically narrowed the CFAA in Van Buren v. United States saying "exceeds authorized access" applies to accessing areas of a system you aren't entitled to enter at all, not misusing access you legitimately have (regardless of ToS violations).
Other CFAA cases have ruled similarly - being legitimately granted access (i.e. signing up for an account, or browsing publicly without logging in, since the site is intended to be available to the public) and then misusing it is not "hacking".
So in the U.S., it's not a computer crime ("hacking"/"breach") to scrape data, and nobody uses the term "data breach" to refer to scraping publicly available data on a public site, except for apparently you.
breezybottom 15 hours ago [-]
You still haven't checked the ToS:
"use automated scripts to collect information from or otherwise interact with the Services" — this covers the entire scraping operation.
"make unauthorised copies, modify, adapt, translate, reverse engineer, disassemble, decompile or create any derivative works of the Services... or determine or attempt to determine any source code" — e.g. reverse-engineering the X-Argus/X-Gorgon/X-Ladon signing scheme.
"interfere with or attempt to interfere with the proper working of the Services... or bypass any measures we may use to prevent or restrict access to the Services" — TLS-fingerprint spoofing, the empty-200 soft block, and the proxy IP rotation to get around rate limiting.
"use or attempt to use another's account, service or system without authorisation from TikTok, or create a false identity on the Services" — this covers the forged device registrations (fake Android handset + carrier profiles)
"use the Services, without our express written consent, for any commercial or unauthorized purpose" — the website is monetizing the dataset and selling the code itself.
edoceo 15 hours ago [-]
Still not a hack or a breach. Still "public" data. Still a TOS violation. Your up-thread claim that potential hack is the same as public is still incorrect.
echoangle 15 hours ago [-]
And who cares about ToS? That’s a purely civil thing, surely hacking (which is a crime) is worse than breaching a contract?
echoangle 15 hours ago [-]
Is it a data breach if I right click images in google search and save them to my disk?
breezybottom 15 hours ago [-]
A data breach could leak into search engine results, sure.
echoangle 15 hours ago [-]
I wasn’t asking if a data breach could be in the search results, I asked if saving search engine results in itself would be a data breach.
Apparently you think that scraping publicly accessible data from tik tok is a data breach in itself, so does this apply in other cases too?
echoangle 15 hours ago [-]
I could get the same data in principle by downloading the app and accessing the post. There’s no hacking or circumventing access controls. That’s what I meant by public data.
arcfour 17 hours ago [-]
No, I don't care at all, but I recognize that my morals might be lower than others here. To me it's just public data...whatever.
My criticism was basically - this is trying to sell an AI slop project for $699 a pop - I could get this out of a few Claude Code sessions if I had the storage and network bandwidth to run such a scraper. The value proposition is questionable when the writing shows that the entire project was AI generated, and clearly Claude understands the way the TikTok Android app internal API works quite well...
arvid-lind 17 hours ago [-]
The AI giants got where they are by pushing similar limits and setting aside moral (and legal) issues to be settled later in court, so the idea seems to fit the general zeitgeist we're living in. Not really criticism, just an observation.
negura 2 hours ago [-]
Very impressive figuring out all 4 checks. I'm not even sure this was reverse engineered, rather than leaked. I wish it mentioned anything about the methodology they used
> Everything described here is a private Go repository. One-time payment, permanent access, complete source.
> $699 one time · lifetime access
Not open-source apparently.
And I cant find the reddit post but I think I read that videos/assets are not actually pre-downloaded, they have to be requested through Tiktok API using the provided code. So if Tiktok patches, the code will need updates too.
nomilk 17 hours ago [-]
I think the post is essentially a decent technical-explainer (value adding and interesting) in exchange for effectively a little product placement (selling either just the code or code already running on a server at additional cost)
Well, most engineers are familiar with GIGO. Garbage in...
Err, thats where my comment stops. Garbage in. Yup.
smallerize 17 hours ago [-]
There's no way this dataset is going to survive on HF, right? It will be hit with so many DMCA takedowns.
simonw 16 hours ago [-]
It's metadata only, not video content. I expect it will likely survive - that's a pretty common pattern for machine learning datasets.
Stable Diffusion was enabled by LAION, for example. That was metadata about images and URLs to those images, but not the actual image files.
smallerize 15 hours ago [-]
True, but it includes the full text of the post. That's enough for some copyright strikes.
deviation 17 hours ago [-]
My thoughts also
magicmicah85 17 hours ago [-]
>Is it legal? It is against TikTok's terms of service. It is sold for research and educational use.
Oh, ok. Otherwise, very detailed deconstruction to scrape their API. Lots of layers of registration and creating a request that looks like it is valid client.
18 hours ago [-]
vachina 16 hours ago [-]
The AI keeps mentioning how a HTTP 200 can silently pollute your dataset. Why not just check contents of body? Usually APIs follow strict JSON contract for successful queries, alert or throw an error when that changes.
405error 16 hours ago [-]
It's probably AI coded and hallucinated many things. That drumming up the importance of a minor thing is a real tell. Another hallucination - it hasn't found any video APIs (despite statements that it has and uploaded it). It has video metadata.
ckugblenu 17 hours ago [-]
This is being posted all over the place. on multiple subreddits and stuff. Why?
vachina 17 hours ago [-]
Ads for an exploit
Retr0id 17 hours ago [-]
Replicating client behaviour is hardly an exploit.
pr337h4m 17 hours ago [-]
There aren't any actual videos in the dataset though.
405error 17 hours ago [-]
The first immediate smell is that if you have 4.5B rows and 289GB in data, you have ~60 bytes per row.
Textbook example of what not to post on the internet. Unreadable incoherent LLM hallucination slop.
igor_nast 17 hours ago [-]
Is this even useful in any way?
negura 2 hours ago [-]
There is even a use case at the end which tracks which songs are currently trending. You basically get access to all of TikTok's data. If you don't see why analytics on this data are so valuable, lookup varoufaki's concept of cloud capital.
smallerize 16 hours ago [-]
Make your own recommendation engine. Even if it's just a real literal text search that sorts chronologically.
tweakimp 16 hours ago [-]
You could use it as negative reinforcement to tell the next big AI model what it should not do
theplumber 16 hours ago [-]
Yes, train your model to give people more AI slop until they get sick of it
megagpt6 17 hours ago [-]
[dead]
028363922092 17 hours ago [-]
[dead]
asadsjaanl 13 hours ago [-]
[flagged]
asadsjaanl 13 hours ago [-]
[flagged]
raver1975 17 hours ago [-]
That data does not belong to the public. Why do you think it is OK to steal from TikTok?
Very LLMish language!
(I guess that is sort of a roundabout summary...)
ToS are just what you follow if you don't want to get banned off of the site. If you don't care about that, then you can go hog wild, though you're being a bit of a jerk/not playing nice obviously.
They might be surprised that you or I looked at their TikTok video when we aren't the intended audience, but they still posted it publicly, with the understanding that it would be made freely available to others.
With how common data hacks are, why wouldn't a reasonable person expect their medical records to leak? I received at least two such breach notices just last year.
If that was not obvious to you then I apologize; though it really should have been, since you are encouraged to interact with others in good faith on HN.
And if you expect your medical records to be public...then what is the point of this discussion?
In hiQ Labs v. LinkedIn, the 9th Circuit (US) ruled that scraping publicly accessible data does not violate the CFAA's "without authorization" clause (hiQ was bulk scraping public LinkedIn profile data - in violation of LinkedIn's ToS). The Supreme Court later specifically narrowed the CFAA in Van Buren v. United States saying "exceeds authorized access" applies to accessing areas of a system you aren't entitled to enter at all, not misusing access you legitimately have (regardless of ToS violations).
Other CFAA cases have ruled similarly - being legitimately granted access (i.e. signing up for an account, or browsing publicly without logging in, since the site is intended to be available to the public) and then misusing it is not "hacking".
So in the U.S., it's not a computer crime ("hacking"/"breach") to scrape data, and nobody uses the term "data breach" to refer to scraping publicly available data on a public site, except for apparently you.
"use automated scripts to collect information from or otherwise interact with the Services" — this covers the entire scraping operation.
"make unauthorised copies, modify, adapt, translate, reverse engineer, disassemble, decompile or create any derivative works of the Services... or determine or attempt to determine any source code" — e.g. reverse-engineering the X-Argus/X-Gorgon/X-Ladon signing scheme.
"interfere with or attempt to interfere with the proper working of the Services... or bypass any measures we may use to prevent or restrict access to the Services" — TLS-fingerprint spoofing, the empty-200 soft block, and the proxy IP rotation to get around rate limiting.
"use or attempt to use another's account, service or system without authorisation from TikTok, or create a false identity on the Services" — this covers the forged device registrations (fake Android handset + carrier profiles)
"use the Services, without our express written consent, for any commercial or unauthorized purpose" — the website is monetizing the dataset and selling the code itself.
My criticism was basically - this is trying to sell an AI slop project for $699 a pop - I could get this out of a few Claude Code sessions if I had the storage and network bandwidth to run such a scraper. The value proposition is questionable when the writing shows that the entire project was AI generated, and clearly Claude understands the way the TikTok Android app internal API works quite well...
Perhaps Google can succeed where others like Meta and X have failed, or perhaps not
https://storage.courtlistener.com/recap/gov.uscourts.cand.46...
Not open-source apparently.
And I cant find the reddit post but I think I read that videos/assets are not actually pre-downloaded, they have to be requested through Tiktok API using the provided code. So if Tiktok patches, the code will need updates too.
But I think this is the 289GB data (free): https://huggingface.co/datasets/kuben-developer/tiktok-video...
Stable Diffusion was enabled by LAION, for example. That was metadata about images and URLs to those images, but not the actual image files.
Oh, ok. Otherwise, very detailed deconstruction to scrape their API. Lots of layers of registration and creating a request that looks like it is valid client.
https://shikigami.dev - more details here