mirror of
https://github.com/imputnet/cobalt.git
synced 2025-01-29 01:38:26 +00:00
Merge branch 'main' into main
This commit is contained in:
ZEREX
GitHub
commit
d31b577c6c
11
README.md
11
README.md
|
|
@ -14,9 +14,13 @@
|
|||
<a href="https://discord.gg/pQPt8HBUPu">
|
||||
💬 community discord server
|
||||
</a>
|
||||
<br/>
|
||||
<a href="https://x.com/justusecobalt">
|
||||
🐦 twitter
|
||||
</a>
|
||||
<a href="https://bsky.app/profile/cobalt.tools">
|
||||
🦋 bluesky
|
||||
</a>
|
||||
</p>
|
||||
<br/>
|
||||
</div>
|
||||
|
|
@ -34,11 +38,10 @@ this monorepo includes source code for api, frontend, and related packages:
|
|||
it also includes documentation in the [docs tree](/docs/):
|
||||
- [cobalt api documentation](/docs/api.md)
|
||||
- [how to run a cobalt instance](/docs/run-an-instance.md)
|
||||
- [how to protect a cobalt instance](/docs/protect-an-instance.md)
|
||||
- [how to configure a cobalt instance for youtube](/docs/configure-for-youtube.md)
|
||||
- [how to protect a cobalt instance](/docs/protect-an-instance.md) (recommended if you host a public instance)
|
||||
|
||||
### thank you
|
||||
cobalt is sponsored by [royalehosting.net](https://royalehosting.net/?partner=cobalt) and the main processing servers are hosted on their network. we really appreciate their kindness and support!
|
||||
cobalt is sponsored by [royalehosting.net](https://royalehosting.net/?partner=cobalt). a part of our infrastructure is hosted on their network. we really appreciate their kindness and support!
|
||||
|
||||
### ethics
|
||||
cobalt is a tool that makes downloading public content easier. it takes **zero liability**.
|
||||
|
|
@ -50,7 +53,7 @@ it can only download free & publicly accessible content.
|
|||
same content can be downloaded via dev tools of any modern web browser.
|
||||
|
||||
### contributing
|
||||
thank you for considering making a contribution to cobalt! please check the [contributing guidelines here](/CONTRIBUTING.md) before making a pull request.
|
||||
if you're considering contributing to cobalt, first of all, thank you! check the [contribution guidelines here](/CONTRIBUTING.md) before getting started, they'll help you do your best right away.
|
||||
|
||||
### licenses
|
||||
for relevant licensing information, see the [api](api/README.md) and [web](web/README.md) READMEs.
|
||||
|
|
|
|||
|
|
@ -11,12 +11,9 @@ we recommend [deploying your own instance](/docs/run-an-instance.md) if you wish
|
|||
|
||||
you can read [the api documentation here](/docs/api.md).
|
||||
|
||||
> [!WARNING]
|
||||
> the v7 public api (/api/json) will be shut down on **november 11th, 2024**.
|
||||
> you can access documentation for it [here](https://github.com/imputnet/cobalt/blob/7/docs/api.md).
|
||||
|
||||
## supported services
|
||||
this list is not final and keeps expanding over time. if support for a service you want is missing, create an issue (or a pull request 👀).
|
||||
this list is not final and keeps expanding over time!
|
||||
if the desired service isn't supported yet, feel free to create an appropriate issue (or a pull request 👀).
|
||||
|
||||
| service | video + audio | only audio | only video | metadata | rich file names |
|
||||
| :-------- | :-----------: | :--------: | :--------: | :------: | :-------------: |
|
||||
|
|
@ -39,12 +36,13 @@ this list is not final and keeps expanding over time. if support for a service y
|
|||
| twitter/x | ✅ | ✅ | ✅ | ➖ | ➖ |
|
||||
| vimeo | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| vk videos & clips | ✅ | ❌ | ✅ | ✅ | ✅ |
|
||||
| xiaohongshu | ✅ | ✅ | ✅ | ➖ | ➖ |
|
||||
| youtube | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
|
||||
| emoji | meaning |
|
||||
| :-----: | :---------------------- |
|
||||
| ✅ | supported |
|
||||
| ➖ | impossible/unreasonable |
|
||||
| ➖ | unreasonable/impossible |
|
||||
| ❌ | not supported |
|
||||
|
||||
### additional notes or features (per service)
|
||||
|
|
@ -71,36 +69,35 @@ as long as you:
|
|||
- provide a link to the license and indicate if changes to the code were made, and
|
||||
- release the code under the **same license**
|
||||
|
||||
## acknowledgements
|
||||
## open source acknowledgements
|
||||
### ffmpeg
|
||||
cobalt heavily relies on ffmpeg for converting and merging media files. it's an absolutely amazing piece of software offered for anyone for free, yet doesn't receive as much credit as it should.
|
||||
cobalt relies on ffmpeg for muxing and encoding media files. ffmpeg is absolutely spectacular and we're privileged to have an ability to use it for free, just like anyone else. we believe it should be way more recognized.
|
||||
|
||||
you can [support ffmpeg here](https://ffmpeg.org/donations.html)!
|
||||
|
||||
#### ffmpeg-static
|
||||
we use [ffmpeg-static](https://github.com/eugeneware/ffmpeg-static) to get binaries for ffmpeg depending on the platform.
|
||||
|
||||
you can support the developer via various methods listed on their github page! (linked above)
|
||||
|
||||
### youtube.js
|
||||
cobalt relies on [youtube.js](https://github.com/LuanRT/YouTube.js) for interacting with the innertube api, it wouldn't have been possible without it.
|
||||
cobalt relies on **[youtube.js](https://github.com/LuanRT/YouTube.js)** for interacting with youtube's innertube api, it wouldn't have been possible without this package.
|
||||
|
||||
you can support the developer via various methods listed on their github page! (linked above)
|
||||
you can support the developer via various methods listed on their github page!
|
||||
(linked above)
|
||||
|
||||
### many others
|
||||
cobalt also depends on:
|
||||
cobalt-api also depends on:
|
||||
|
||||
- [content-disposition-header](https://www.npmjs.com/package/content-disposition-header) to simplify the provision of `content-disposition` headers.
|
||||
- [cors](https://www.npmjs.com/package/cors) to manage cross-origin resource sharing within expressjs.
|
||||
- [dotenv](https://www.npmjs.com/package/dotenv) to load environment variables from the `.env` file.
|
||||
- [express](https://www.npmjs.com/package/express) as the backbone of cobalt servers.
|
||||
- [express-rate-limit](https://www.npmjs.com/package/express-rate-limit) to rate limit api endpoints.
|
||||
- [hls-parser](https://www.npmjs.com/package/hls-parser) to parse `m3u8` playlists for certain services.
|
||||
- [ipaddr.js](https://www.npmjs.com/package/ipaddr.js) to parse ip addresses (for rate limiting).
|
||||
- [nanoid](https://www.npmjs.com/package/nanoid) to generate unique (temporary) identifiers for each requested stream.
|
||||
- [psl](https://www.npmjs.com/package/psl) as the domain name parser.
|
||||
- [set-cookie-parser](https://www.npmjs.com/package/set-cookie-parser) to parse cookies that cobalt receives from certain services.
|
||||
- [undici](https://www.npmjs.com/package/undici) for making http requests.
|
||||
- [url-pattern](https://www.npmjs.com/package/url-pattern) to match provided links with supported patterns.
|
||||
- **[content-disposition-header](https://www.npmjs.com/package/content-disposition-header)** to simplify the provision of `content-disposition` headers.
|
||||
- **[cors](https://www.npmjs.com/package/cors)** to manage cross-origin resource sharing within expressjs.
|
||||
- **[dotenv](https://www.npmjs.com/package/dotenv)** to load environment variables from the `.env` file.
|
||||
- **[express](https://www.npmjs.com/package/express)** as the backbone of cobalt servers.
|
||||
- **[express-rate-limit](https://www.npmjs.com/package/express-rate-limit)** to rate limit api endpoints.
|
||||
- **[ffmpeg-static](https://www.npmjs.com/package/ffmpeg-static)** to get binaries for ffmpeg depending on the platform.
|
||||
- **[hls-parser](https://www.npmjs.com/package/hls-parser)** to parse HLS playlists according to spec (very impressive stuff).
|
||||
- **[ipaddr.js](https://www.npmjs.com/package/ipaddr.js)** to parse ip addresses (used for rate limiting).
|
||||
- **[nanoid](https://www.npmjs.com/package/nanoid)** to generate unique identifiers for each requested tunnel.
|
||||
- **[set-cookie-parser](https://www.npmjs.com/package/set-cookie-parser)** to parse cookies that cobalt receives from certain services.
|
||||
- **[undici](https://www.npmjs.com/package/undici)** for making http requests.
|
||||
- **[url-pattern](https://www.npmjs.com/package/url-pattern)** to match provided links with supported patterns.
|
||||
- **[zod](https://www.npmjs.com/package/zod)** to lock down the api request schema.
|
||||
- **[@datastructures-js/priority-queue](https://www.npmjs.com/package/@datastructures-js/priority-queue)** for sorting stream caches for future clean up (without redis).
|
||||
- **[@imput/psl](https://www.npmjs.com/package/@imput/psl)** as the domain name parser, our fork of [psl](https://www.npmjs.com/package/psl).
|
||||
|
||||
...and many other packages that these packages rely on.
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"name": "@imput/cobalt-api",
|
||||
"description": "save what you love",
|
||||
"version": "10.5.4",
|
||||
"version": "10.6",
|
||||
"author": "imput",
|
||||
"exports": "./src/cobalt.js",
|
||||
"type": "module",
|
||||
|
|
@ -39,7 +39,7 @@
|
|||
"set-cookie-parser": "2.6.0",
|
||||
"undici": "^5.19.1",
|
||||
"url-pattern": "1.0.3",
|
||||
"youtubei.js": "^12.2.0",
|
||||
"youtubei.js": "^13.0.0",
|
||||
"zod": "^3.23.8"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
|
|
|
|||
|
|
@ -313,7 +313,7 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
|
|||
...Object.entries(req.headers)
|
||||
]);
|
||||
|
||||
return stream(res, { type: 'internal', ...streamInfo });
|
||||
return stream(res, { type: 'internal', data: streamInfo });
|
||||
};
|
||||
|
||||
app.get('/itunnel', itunnelHandler);
|
||||
|
|
|
|||
|
|
@ -1,8 +1,16 @@
|
|||
export function getRedirectingURL(url) {
|
||||
return fetch(url, { redirect: 'manual' }).then((r) => {
|
||||
if ([301, 302, 303].includes(r.status) && r.headers.has('location'))
|
||||
const redirectStatuses = new Set([301, 302, 303, 307, 308]);
|
||||
|
||||
export async function getRedirectingURL(url, dispatcher) {
|
||||
const location = await fetch(url, {
|
||||
redirect: 'manual',
|
||||
dispatcher,
|
||||
}).then((r) => {
|
||||
if (redirectStatuses.has(r.status) && r.headers.has('location')) {
|
||||
return r.headers.get('location');
|
||||
}
|
||||
}).catch(() => null);
|
||||
|
||||
return location;
|
||||
}
|
||||
|
||||
export function merge(a, b) {
|
||||
|
|
@ -29,3 +37,7 @@ export function splitFilenameExtension(filename) {
|
|||
return [ parts.join('.'), ext ]
|
||||
}
|
||||
}
|
||||
|
||||
export function zip(a, b) {
|
||||
return a.map((value, i) => [ value, b[i] ]);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -15,7 +15,8 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
|
|||
filename: r.filenameAttributes ?
|
||||
createFilename(r.filenameAttributes, filenameStyle, isAudioOnly, isAudioMuted) : r.filename,
|
||||
fileMetadata: !disableMetadata ? r.fileMetadata : false,
|
||||
requestIP
|
||||
requestIP,
|
||||
originalRequest: r.originalRequest
|
||||
},
|
||||
params = {};
|
||||
|
||||
|
|
@ -47,7 +48,7 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
|
|||
});
|
||||
|
||||
case "photo":
|
||||
responseType = "redirect";
|
||||
params = { type: "proxy" };
|
||||
break;
|
||||
|
||||
case "gif":
|
||||
|
|
@ -83,6 +84,7 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
|
|||
case "twitter":
|
||||
case "snapchat":
|
||||
case "bsky":
|
||||
case "xiaohongshu":
|
||||
params = { picker: r.picker };
|
||||
break;
|
||||
|
||||
|
|
@ -143,6 +145,7 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
|
|||
case "ok":
|
||||
case "vk":
|
||||
case "tiktok":
|
||||
case "xiaohongshu":
|
||||
params = { type: "proxy" };
|
||||
break;
|
||||
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ import snapchat from "./services/snapchat.js";
|
|||
import loom from "./services/loom.js";
|
||||
import facebook from "./services/facebook.js";
|
||||
import bluesky from "./services/bluesky.js";
|
||||
import xiaohongshu from "./services/xiaohongshu.js";
|
||||
|
||||
let freebind;
|
||||
|
||||
|
|
@ -239,6 +240,15 @@ export default async function({ host, patternMatch, params }) {
|
|||
});
|
||||
break;
|
||||
|
||||
case "xiaohongshu":
|
||||
r = await xiaohongshu({
|
||||
...patternMatch,
|
||||
h265: params.tiktokH265,
|
||||
isAudioOnly,
|
||||
dispatcher,
|
||||
});
|
||||
break;
|
||||
|
||||
default:
|
||||
return createResponse("error", {
|
||||
code: "error.api.service.unsupported"
|
||||
|
|
|
|||
|
|
@ -166,6 +166,14 @@ export const services = {
|
|||
subdomains: ["m"],
|
||||
altDomains: ["vkvideo.ru", "vk.ru"],
|
||||
},
|
||||
xiaohongshu: {
|
||||
patterns: [
|
||||
"explore/:id?xsec_token=:token",
|
||||
"discovery/item/:id?xsec_token=:token",
|
||||
"a/:shareId"
|
||||
],
|
||||
altDomains: ["xhslink.com"],
|
||||
},
|
||||
youtube: {
|
||||
patterns: [
|
||||
"watch?v=:id",
|
||||
|
|
|
|||
|
|
@ -71,4 +71,8 @@ export const testers = {
|
|||
|
||||
"bsky": pattern =>
|
||||
pattern.user?.length <= 128 && pattern.post?.length <= 128,
|
||||
|
||||
"xiaohongshu": pattern =>
|
||||
pattern.id?.length <= 24 && pattern.token?.length <= 64
|
||||
|| pattern.shareId?.length <= 12,
|
||||
}
|
||||
|
|
|
|||
|
|
@ -71,6 +71,24 @@ const extractImages = ({ getPost, filename, alwaysProxy }) => {
|
|||
return { picker };
|
||||
}
|
||||
|
||||
const extractGif = ({ url, filename }) => {
|
||||
const gifUrl = new URL(url);
|
||||
|
||||
if (!gifUrl || gifUrl.hostname !== "media.tenor.com") {
|
||||
return { error: "fetch.empty" };
|
||||
}
|
||||
|
||||
// remove downscaling params from gif url
|
||||
// such as "?hh=498&ww=498"
|
||||
gifUrl.search = "";
|
||||
|
||||
return {
|
||||
urls: gifUrl,
|
||||
isPhoto: true,
|
||||
filename: `${filename}.gif`,
|
||||
}
|
||||
}
|
||||
|
||||
export default async function ({ user, post, alwaysProxy, dispatcher }) {
|
||||
const apiEndpoint = new URL("https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread?depth=0&parentHeight=0");
|
||||
apiEndpoint.searchParams.set(
|
||||
|
|
@ -102,22 +120,37 @@ export default async function ({ user, post, alwaysProxy, dispatcher }) {
|
|||
const embedType = getPost?.thread?.post?.embed?.$type;
|
||||
const filename = `bluesky_${user}_${post}`;
|
||||
|
||||
if (embedType === "app.bsky.embed.video#view") {
|
||||
return extractVideo({
|
||||
media: getPost.thread?.post?.embed,
|
||||
filename,
|
||||
})
|
||||
}
|
||||
switch (embedType) {
|
||||
case "app.bsky.embed.video#view":
|
||||
return extractVideo({
|
||||
media: getPost.thread?.post?.embed,
|
||||
filename,
|
||||
});
|
||||
|
||||
if (embedType === "app.bsky.embed.recordWithMedia#view") {
|
||||
return extractVideo({
|
||||
media: getPost.thread?.post?.embed?.media,
|
||||
filename,
|
||||
})
|
||||
}
|
||||
case "app.bsky.embed.images#view":
|
||||
return extractImages({
|
||||
getPost,
|
||||
filename,
|
||||
alwaysProxy
|
||||
});
|
||||
|
||||
if (embedType === "app.bsky.embed.images#view") {
|
||||
return extractImages({ getPost, filename, alwaysProxy });
|
||||
case "app.bsky.embed.external#view":
|
||||
return extractGif({
|
||||
url: getPost?.thread?.post?.embed?.external?.uri,
|
||||
filename,
|
||||
});
|
||||
|
||||
case "app.bsky.embed.recordWithMedia#view":
|
||||
if (getPost?.thread?.post?.embed?.media?.$type === "app.bsky.embed.external#view") {
|
||||
return extractGif({
|
||||
url: getPost?.thread?.post?.embed?.media?.external?.uri,
|
||||
filename,
|
||||
});
|
||||
}
|
||||
return extractVideo({
|
||||
media: getPost.thread?.post?.embed?.media,
|
||||
filename,
|
||||
});
|
||||
}
|
||||
|
||||
return { error: "fetch.empty" };
|
||||
|
|
|
|||
116
api/src/processing/services/xiaohongshu.js
Normal file
116
api/src/processing/services/xiaohongshu.js
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
import { extract, normalizeURL } from "../url.js";
|
||||
import { genericUserAgent } from "../../config.js";
|
||||
import { createStream } from "../../stream/manage.js";
|
||||
import { getRedirectingURL } from "../../misc/utils.js";
|
||||
|
||||
const https = (url) => {
|
||||
return url.replace(/^http:/i, 'https:');
|
||||
}
|
||||
|
||||
export default async function ({ id, token, shareId, h265, isAudioOnly, dispatcher }) {
|
||||
let noteId = id;
|
||||
let xsecToken = token;
|
||||
|
||||
if (!noteId) {
|
||||
const extractedURL = await getRedirectingURL(
|
||||
`https://xhslink.com/a/${shareId}`,
|
||||
dispatcher
|
||||
);
|
||||
|
||||
if (extractedURL) {
|
||||
const { patternMatch } = extract(normalizeURL(extractedURL));
|
||||
|
||||
if (patternMatch) {
|
||||
noteId = patternMatch.id;
|
||||
xsecToken = patternMatch.token;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!noteId || !xsecToken) return { error: "fetch.short_link" };
|
||||
|
||||
const res = await fetch(`https://www.xiaohongshu.com/explore/${noteId}?xsec_token=${xsecToken}`, {
|
||||
headers: {
|
||||
"user-agent": genericUserAgent,
|
||||
},
|
||||
dispatcher,
|
||||
});
|
||||
|
||||
const html = await res.text();
|
||||
|
||||
let note;
|
||||
try {
|
||||
const initialState = html
|
||||
.split('<script>window.__INITIAL_STATE__=')[1]
|
||||
.split('</script>')[0]
|
||||
.replace(/:\s*undefined/g, ":null");
|
||||
|
||||
const data = JSON.parse(initialState);
|
||||
|
||||
const noteInfo = data?.note?.noteDetailMap;
|
||||
if (!noteInfo) throw "no note detail map";
|
||||
|
||||
const currentNote = noteInfo[noteId];
|
||||
if (!currentNote) throw "no current note in detail map";
|
||||
|
||||
note = currentNote.note;
|
||||
} catch {}
|
||||
|
||||
if (!note) return { error: "fetch.empty" };
|
||||
|
||||
const video = note.video;
|
||||
const images = note.imageList;
|
||||
|
||||
const filenameBase = `xiaohongshu_${noteId}`;
|
||||
|
||||
if (video) {
|
||||
const videoFilename = `${filenameBase}.mp4`;
|
||||
const audioFilename = `${filenameBase}_audio`;
|
||||
|
||||
let videoURL;
|
||||
|
||||
if (h265 && !isAudioOnly && video.consumer?.originVideoKey) {
|
||||
videoURL = `https://sns-video-bd.xhscdn.com/${video.consumer.originVideoKey}`;
|
||||
} else {
|
||||
const h264Streams = video.media?.stream?.h264;
|
||||
|
||||
if (h264Streams?.length) {
|
||||
videoURL = h264Streams.reduce((a, b) => Number(a?.videoBitrate) > Number(b?.videoBitrate) ? a : b).masterUrl;
|
||||
}
|
||||
}
|
||||
|
||||
if (!videoURL) return { error: "fetch.empty" };
|
||||
|
||||
return {
|
||||
urls: https(videoURL),
|
||||
filename: videoFilename,
|
||||
audioFilename: audioFilename,
|
||||
}
|
||||
}
|
||||
|
||||
if (!images || images.length === 0) {
|
||||
return { error: "fetch.empty" };
|
||||
}
|
||||
|
||||
if (images.length === 1) {
|
||||
return {
|
||||
isPhoto: true,
|
||||
urls: https(images[0].urlDefault),
|
||||
filename: `${filenameBase}.jpg`,
|
||||
}
|
||||
}
|
||||
|
||||
const picker = images.map((image, i) => {
|
||||
return {
|
||||
type: "photo",
|
||||
url: createStream({
|
||||
service: "xiaohongshu",
|
||||
type: "proxy",
|
||||
url: https(image.urlDefault),
|
||||
filename: `${filenameBase}_${i + 1}.jpg`,
|
||||
})
|
||||
}
|
||||
});
|
||||
|
||||
return { picker };
|
||||
}
|
||||
|
|
@ -149,7 +149,7 @@ export default async function (o) {
|
|||
useHLS = false;
|
||||
}
|
||||
|
||||
let innertubeClient = "ANDROID";
|
||||
let innertubeClient = o.innertubeClient || "ANDROID";
|
||||
|
||||
if (cookie) {
|
||||
useHLS = false;
|
||||
|
|
@ -240,12 +240,12 @@ export default async function (o) {
|
|||
const quality = o.quality === "max" ? 9000 : Number(o.quality);
|
||||
|
||||
const normalizeQuality = res => {
|
||||
const shortestSide = res.height > res.width ? res.width : res.height;
|
||||
const shortestSide = Math.min(res.height, res.width);
|
||||
return videoQualities.find(qual => qual >= shortestSide);
|
||||
}
|
||||
|
||||
let video, audio, dubbedLanguage,
|
||||
codec = o.format || "h264";
|
||||
codec = o.format || "h264", itag = o.itag;
|
||||
|
||||
if (useHLS) {
|
||||
const hlsManifest = info.streaming_data.hls_manifest_url;
|
||||
|
|
@ -351,17 +351,21 @@ export default async function (o) {
|
|||
Number(b.bitrate) - Number(a.bitrate)
|
||||
).forEach(format => {
|
||||
Object.keys(codecList).forEach(yCodec => {
|
||||
const matchingItag = slot => !itag?.[slot] || itag[slot] === format.itag;
|
||||
const sorted = sorted_formats[yCodec];
|
||||
const goodFormat = checkFormat(format, yCodec);
|
||||
if (!goodFormat) return;
|
||||
|
||||
if (format.has_video) {
|
||||
if (format.has_video && matchingItag('video')) {
|
||||
sorted.video.push(format);
|
||||
if (!sorted.bestVideo) sorted.bestVideo = format;
|
||||
if (!sorted.bestVideo)
|
||||
sorted.bestVideo = format;
|
||||
}
|
||||
if (format.has_audio) {
|
||||
|
||||
if (format.has_audio && matchingItag('audio')) {
|
||||
sorted.audio.push(format);
|
||||
if (!sorted.bestAudio) sorted.bestAudio = format;
|
||||
if (!sorted.bestAudio)
|
||||
sorted.bestAudio = format;
|
||||
}
|
||||
})
|
||||
});
|
||||
|
|
@ -448,6 +452,18 @@ export default async function (o) {
|
|||
youtubeDubName: dubbedLanguage || false,
|
||||
}
|
||||
|
||||
itag = {
|
||||
video: video?.itag,
|
||||
audio: audio?.itag
|
||||
};
|
||||
|
||||
const originalRequest = {
|
||||
...o,
|
||||
dispatcher: undefined,
|
||||
itag,
|
||||
innertubeClient
|
||||
};
|
||||
|
||||
if (audio && o.isAudioOnly) {
|
||||
let bestAudio = codec === "h264" ? "m4a" : "opus";
|
||||
let urls = audio.url;
|
||||
|
|
@ -469,6 +485,7 @@ export default async function (o) {
|
|||
fileMetadata,
|
||||
bestAudio,
|
||||
isHLS: useHLS,
|
||||
originalRequest
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -494,7 +511,7 @@ export default async function (o) {
|
|||
if (innertubeClient === "WEB_EMBEDDED" && innertube) {
|
||||
video = video.decipher(innertube.session.player);
|
||||
audio = audio.decipher(innertube.session.player);
|
||||
} else if (video.url && audio.url) {
|
||||
} else {
|
||||
video = video.url;
|
||||
audio = audio.url;
|
||||
}
|
||||
|
|
@ -512,6 +529,7 @@ export default async function (o) {
|
|||
filenameAttributes,
|
||||
fileMetadata,
|
||||
isHLS: useHLS,
|
||||
originalRequest
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -92,9 +92,14 @@ function aliasURL(url) {
|
|||
url.hostname = 'vk.com';
|
||||
}
|
||||
break;
|
||||
|
||||
case "xhslink":
|
||||
if (url.hostname === 'xhslink.com' && parts.length === 3) {
|
||||
url = new URL(`https://www.xiaohongshu.com/a/${parts[2]}`);
|
||||
}
|
||||
}
|
||||
|
||||
return url
|
||||
return url;
|
||||
}
|
||||
|
||||
function cleanURL(url) {
|
||||
|
|
@ -114,36 +119,41 @@ function cleanURL(url) {
|
|||
break;
|
||||
case "vk":
|
||||
if (url.pathname.includes('/clip') && url.searchParams.get('z')) {
|
||||
limitQuery('z')
|
||||
limitQuery('z');
|
||||
}
|
||||
break;
|
||||
case "youtube":
|
||||
if (url.searchParams.get('v')) {
|
||||
limitQuery('v')
|
||||
limitQuery('v');
|
||||
}
|
||||
break;
|
||||
case "rutube":
|
||||
if (url.searchParams.get('p')) {
|
||||
limitQuery('p')
|
||||
limitQuery('p');
|
||||
}
|
||||
break;
|
||||
case "twitter":
|
||||
if (url.searchParams.get('post_id')) {
|
||||
limitQuery('post_id')
|
||||
limitQuery('post_id');
|
||||
}
|
||||
break;
|
||||
case "xiaohongshu":
|
||||
if (url.searchParams.get('xsec_token')) {
|
||||
limitQuery('xsec_token');
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
if (stripQuery) {
|
||||
url.search = ''
|
||||
url.search = '';
|
||||
}
|
||||
|
||||
url.username = url.password = url.port = url.hash = ''
|
||||
url.username = url.password = url.port = url.hash = '';
|
||||
|
||||
if (url.pathname.endsWith('/'))
|
||||
url.pathname = url.pathname.slice(0, -1);
|
||||
|
||||
return url
|
||||
return url;
|
||||
}
|
||||
|
||||
function getHostIfValid(url) {
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ const CHUNK_SIZE = BigInt(8e6); // 8 MB
|
|||
const min = (a, b) => a < b ? a : b;
|
||||
|
||||
async function* readChunks(streamInfo, size) {
|
||||
let read = 0n;
|
||||
let read = 0n, chunksSinceTransplant = 0;
|
||||
while (read < size) {
|
||||
if (streamInfo.controller.signal.aborted) {
|
||||
throw new Error("controller aborted");
|
||||
|
|
@ -22,6 +22,16 @@ async function* readChunks(streamInfo, size) {
|
|||
signal: streamInfo.controller.signal
|
||||
});
|
||||
|
||||
if (chunk.statusCode === 403 && chunksSinceTransplant >= 3 && streamInfo.transplant) {
|
||||
chunksSinceTransplant = 0;
|
||||
try {
|
||||
await streamInfo.transplant(streamInfo.dispatcher);
|
||||
continue;
|
||||
} catch {}
|
||||
}
|
||||
|
||||
chunksSinceTransplant++;
|
||||
|
||||
const expected = min(CHUNK_SIZE, size - read);
|
||||
const received = BigInt(chunk.headers['content-length']);
|
||||
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import { env } from "../config.js";
|
|||
import { closeRequest } from "./shared.js";
|
||||
import { decryptStream, encryptStream } from "../misc/crypto.js";
|
||||
import { hashHmac } from "../security/secrets.js";
|
||||
import { zip } from "../misc/utils.js";
|
||||
|
||||
// optional dependency
|
||||
const freebind = env.freebindCIDR && await import('freebind').catch(() => {});
|
||||
|
|
@ -40,6 +41,7 @@ export function createStream(obj) {
|
|||
audioFormat: obj.audioFormat,
|
||||
|
||||
isHLS: obj.isHLS || false,
|
||||
originalRequest: obj.originalRequest
|
||||
};
|
||||
|
||||
// FIXME: this is now a Promise, but it is not awaited
|
||||
|
|
@ -100,6 +102,7 @@ export function createInternalStream(url, obj = {}) {
|
|||
controller,
|
||||
dispatcher,
|
||||
isHLS: obj.isHLS,
|
||||
transplant: obj.transplant
|
||||
});
|
||||
|
||||
let streamLink = new URL('/itunnel', `http://127.0.0.1:${env.tunnelPort}`);
|
||||
|
|
@ -115,13 +118,17 @@ export function createInternalStream(url, obj = {}) {
|
|||
return streamLink.toString();
|
||||
}
|
||||
|
||||
export function destroyInternalStream(url) {
|
||||
function getInternalTunnelId(url) {
|
||||
url = new URL(url);
|
||||
if (url.hostname !== '127.0.0.1') {
|
||||
return;
|
||||
}
|
||||
|
||||
const id = url.searchParams.get('id');
|
||||
return url.searchParams.get('id');
|
||||
}
|
||||
|
||||
export function destroyInternalStream(url) {
|
||||
const id = getInternalTunnelId(url);
|
||||
|
||||
if (internalStreamCache.has(id)) {
|
||||
closeRequest(getInternalStream(id)?.controller);
|
||||
|
|
@ -129,9 +136,68 @@ export function destroyInternalStream(url) {
|
|||
}
|
||||
}
|
||||
|
||||
const transplantInternalTunnels = function(tunnelUrls, transplantUrls) {
|
||||
if (tunnelUrls.length !== transplantUrls.length) {
|
||||
return;
|
||||
}
|
||||
|
||||
for (const [ tun, url ] of zip(tunnelUrls, transplantUrls)) {
|
||||
const id = getInternalTunnelId(tun);
|
||||
const itunnel = getInternalStream(id);
|
||||
|
||||
if (!itunnel) continue;
|
||||
itunnel.url = url;
|
||||
}
|
||||
}
|
||||
|
||||
const transplantTunnel = async function (dispatcher) {
|
||||
if (this.pendingTransplant) {
|
||||
await this.pendingTransplant;
|
||||
return;
|
||||
}
|
||||
|
||||
let finished;
|
||||
this.pendingTransplant = new Promise(r => finished = r);
|
||||
|
||||
try {
|
||||
const handler = await import(`../processing/services/${this.service}.js`);
|
||||
const response = await handler.default({
|
||||
...this.originalRequest,
|
||||
dispatcher
|
||||
});
|
||||
|
||||
if (!response.urls) {
|
||||
return;
|
||||
}
|
||||
|
||||
response.urls = [response.urls].flat();
|
||||
if (this.originalRequest.isAudioOnly && response.urls.length > 1) {
|
||||
response.urls = [response.urls[1]];
|
||||
} else if (this.originalRequest.isAudioMuted) {
|
||||
response.urls = [response.urls[0]];
|
||||
}
|
||||
|
||||
const tunnels = [this.urls].flat();
|
||||
if (tunnels.length !== response.urls.length) {
|
||||
return;
|
||||
}
|
||||
|
||||
transplantInternalTunnels(tunnels, response.urls);
|
||||
}
|
||||
catch {}
|
||||
finally {
|
||||
finished();
|
||||
delete this.pendingTransplant;
|
||||
}
|
||||
}
|
||||
|
||||
function wrapStream(streamInfo) {
|
||||
const url = streamInfo.urls;
|
||||
|
||||
if (streamInfo.originalRequest) {
|
||||
streamInfo.transplant = transplantTunnel.bind(streamInfo);
|
||||
}
|
||||
|
||||
if (typeof url === 'string') {
|
||||
streamInfo.urls = createInternalStream(url, streamInfo);
|
||||
} else if (Array.isArray(url)) {
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ export default async function(res, streamInfo) {
|
|||
return await stream.proxy(streamInfo, res);
|
||||
|
||||
case "internal":
|
||||
return internalStream(streamInfo, res);
|
||||
return internalStream(streamInfo.data, res);
|
||||
|
||||
case "merge":
|
||||
return stream.merge(streamInfo, res);
|
||||
|
|
|
|||
|
|
@ -54,7 +54,25 @@
|
|||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "redirect"
|
||||
"status": "tunnel"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "gif with a quoted post",
|
||||
"url": "https://bsky.app/profile/imlunahey.com/post/3lgajpn5dtk2t",
|
||||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "tunnel"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "gif alone in a post",
|
||||
"url": "https://bsky.app/profile/imlunahey.com/post/3lgah3ovxnc2q",
|
||||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "tunnel"
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
|
|||
|
|
@ -54,7 +54,7 @@
|
|||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "redirect"
|
||||
"status": "tunnel"
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -63,7 +63,7 @@
|
|||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "redirect"
|
||||
"status": "tunnel"
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -72,7 +72,7 @@
|
|||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "redirect"
|
||||
"status": "tunnel"
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -81,7 +81,7 @@
|
|||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "redirect"
|
||||
"status": "tunnel"
|
||||
}
|
||||
}
|
||||
]
|
||||
58
api/src/util/tests/xiaohongshu.json
Normal file
58
api/src/util/tests/xiaohongshu.json
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
[
|
||||
{
|
||||
"name": "long link video",
|
||||
"url": "https://www.xiaohongshu.com/discovery/item/6789065900000000210035fc?source=webshare&xhsshare=pc_web&xsec_token=CBustnz_Twf1BSybpe5-D-BzUb-Bx28DPLb418TN9S9Kk&xsec_source=pc_share",
|
||||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "tunnel"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "picker with multiple live photos",
|
||||
"url": "https://www.xiaohongshu.com/explore/67847fa1000000000203e6ed?xsec_token=CBzyP7Y44PPpsM20lgxqrIIJMHqOLemusDsRcmsX0cTpk",
|
||||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "picker"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "one photo",
|
||||
"url": "https://www.xiaohongshu.com/explore/6788b56200000000210008c8?xsec_token=CBSDiWU4N-DgirHrOVbIWrlKfUNFHKwm-Wsjqz7dIMc_k",
|
||||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "tunnel"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "short link, might expire eventually",
|
||||
"url": "https://xhslink.com/a/czn4z6c1tic4",
|
||||
"canFail": true,
|
||||
"params": {},
|
||||
"expected": {
|
||||
"code": 200,
|
||||
"status": "tunnel"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "wrong note id",
|
||||
"url": "https://www.xiaohongshu.com/discovery/item/6789065911100000210035fc?source=webshare&xhsshare=pc_web&xsec_token=CBustnz_Twf1BSybpe5-D-BzUb-Bx28DPLb418TN9S9Kk&xsec_source=pc_share",
|
||||
"params": {},
|
||||
"expected": {
|
||||
"code": 400,
|
||||
"status": "error"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "short link, wrong id",
|
||||
"url": "https://xhslink.com/a/aaaaaa",
|
||||
"canFail": true,
|
||||
"params": {},
|
||||
"expected": {
|
||||
"code": 400,
|
||||
"status": "error"
|
||||
}
|
||||
}
|
||||
]
|
||||
|
|
@ -68,7 +68,7 @@ Content-Type: application/json
|
|||
| `alwaysProxy` | `boolean` | `true / false` | `false` | tunnels all downloads through the processing server, even when not necessary. |
|
||||
| `disableMetadata` | `boolean` | `true / false` | `false` | disables file metadata when set to `true`. |
|
||||
| `tiktokFullAudio` | `boolean` | `true / false` | `false` | enables download of original sound used in a tiktok video. |
|
||||
| `tiktokH265` | `boolean` | `true / false` | `false` | changes whether 1080p h265 videos are preferred or not. |
|
||||
| `tiktokH265` | `boolean` | `true / false` | `false` | allows h265 videos when enabled. applies to tiktok & xiaohongshu. |
|
||||
| `twitterGif` | `boolean` | `true / false` | `true` | changes whether twitter gifs are converted to .gif |
|
||||
| `youtubeHLS` | `boolean` | `true / false` | `false` | specifies whether to use HLS for downloading video or audio from youtube. |
|
||||
|
||||
|
|
|
|||
|
|
@ -56,8 +56,8 @@ importers:
|
|||
specifier: 1.0.3
|
||||
version: 1.0.3
|
||||
youtubei.js:
|
||||
specifier: ^12.2.0
|
||||
version: 12.2.0
|
||||
specifier: ^13.0.0
|
||||
version: 13.0.0
|
||||
zod:
|
||||
specifier: ^3.23.8
|
||||
version: 3.23.8
|
||||
|
|
@ -2286,8 +2286,8 @@ packages:
|
|||
resolution: {integrity: sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==}
|
||||
engines: {node: '>=10'}
|
||||
|
||||
youtubei.js@12.2.0:
|
||||
resolution: {integrity: sha512-G+50qrbJCToMYhu8jbaHiS3Vf+RRul+CcDbz3hEGwHkGPh+zLiWwD6SS+YhYF+2/op4ZU5zDYQJrGqJ+wKh7Gw==}
|
||||
youtubei.js@13.0.0:
|
||||
resolution: {integrity: sha512-b1QkN9bfgphK+5tI4qteSK54kNxmPhoedvMw0jl4uSn+L8gbDbJ4z52amNuYNcOdp4X/SI3JuUb+f5V0DPJ8Vw==}
|
||||
|
||||
zod@3.23.8:
|
||||
resolution: {integrity: sha512-XBx9AXhXktjUqnepgTiE5flcKIYWi/rme0Eaj+5Y0lftuGBq+jyRu/md4WnuxqgP1ubdpNCsYEYPxrzVHD8d6g==}
|
||||
|
|
@ -4242,7 +4242,7 @@ snapshots:
|
|||
|
||||
yocto-queue@0.1.0: {}
|
||||
|
||||
youtubei.js@12.2.0:
|
||||
youtubei.js@13.0.0:
|
||||
dependencies:
|
||||
'@bufbuild/protobuf': 2.1.0
|
||||
jintr: 3.2.0
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ no ads, trackers, paywalls, or other nonsense. just a convenient web app that wo
|
|||
cobalt was created for public benefit, to protect people from ads and malware pushed by its alternatives.
|
||||
we believe that the best software is safe, open, and accessible.
|
||||
|
||||
it's possible to keep the main instances up thanks to our long-standing infrastructure partner, [royalehosting.net]({partners.royalehosting})!
|
||||
a part of our infrastructure is provided by our long-standing partner, [royalehosting.net]({partners.royalehosting})!
|
||||
</section>
|
||||
|
||||
<section id="privacy">
|
||||
|
|
|
|||
|
|
@ -40,9 +40,9 @@
|
|||
"video.twitter.gif.title": "convert looping videos to GIF",
|
||||
"video.twitter.gif.description": "GIF conversion is inefficient, converted file may be obnoxiously big and low quality.",
|
||||
|
||||
"video.tiktok.h265": "tiktok",
|
||||
"video.tiktok.h265.title": "prefer HEVC/H265 format",
|
||||
"video.tiktok.h265.description": "allows downloading videos in 1080p at cost of compatibility.",
|
||||
"video.h265": "high efficiency video codec",
|
||||
"video.h265.title": "allow h265 for videos",
|
||||
"video.h265.description": "allows downloading videos from platforms like tiktok and xiaohongshu in higher quality at cost of compatibility.",
|
||||
|
||||
"audio.format": "audio format",
|
||||
"audio.format.best": "best",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "@imput/cobalt-web",
|
||||
"version": "10.5.1",
|
||||
"version": "10.6",
|
||||
"type": "module",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@
|
|||
import dialogs from "$lib/state/dialogs";
|
||||
import { link } from "$lib/state/omnibox";
|
||||
import { updateSetting } from "$lib/state/settings";
|
||||
import { pasteLinkFromClipboard } from "$lib/clipboard";
|
||||
import { turnstileEnabled, turnstileSolved } from "$lib/state/turnstile";
|
||||
|
||||
import type { Optional } from "$lib/types/generic";
|
||||
|
|
@ -41,7 +42,7 @@
|
|||
|
||||
const validLink = (url: string) => {
|
||||
try {
|
||||
return /^https:/i.test(new URL(url).protocol);
|
||||
return /^https?\:/i.test(new URL(url).protocol);
|
||||
} catch {}
|
||||
};
|
||||
|
||||
|
|
@ -59,22 +60,24 @@
|
|||
goto("/", { replaceState: true });
|
||||
}
|
||||
|
||||
const pasteClipboard = () => {
|
||||
const pasteClipboard = async () => {
|
||||
if ($dialogs.length > 0 || isDisabled || isLoading) {
|
||||
return;
|
||||
}
|
||||
|
||||
navigator.clipboard.readText().then(async (text: string) => {
|
||||
let matchLink = text.match(/https:\/\/[^\s]+/g);
|
||||
if (matchLink) {
|
||||
$link = matchLink[0];
|
||||
const pastedData = await pasteLinkFromClipboard();
|
||||
if (!pastedData) return;
|
||||
|
||||
if (!isBotCheckOngoing) {
|
||||
await tick(); // wait for button to render
|
||||
downloadButton.download($link);
|
||||
}
|
||||
const linkMatch = pastedData.match(/https?\:\/\/[^\s]+/g);
|
||||
|
||||
if (linkMatch) {
|
||||
$link = linkMatch[0].split('，')[0];
|
||||
|
||||
if (!isBotCheckOngoing) {
|
||||
await tick(); // wait for button to render
|
||||
downloadButton.download($link);
|
||||
}
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
const changeDownloadMode = (mode: DownloadModeOption) => {
|
||||
|
|
|
|||
17
web/src/lib/clipboard.ts
Normal file
17
web/src/lib/clipboard.ts
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
const allowedLinkTypes = new Set(["text/plain", "text/uri-list"]);
|
||||
|
||||
export const pasteLinkFromClipboard = async () => {
|
||||
const clipboard = await navigator.clipboard.read();
|
||||
|
||||
if (clipboard?.length) {
|
||||
const clipboardItem = clipboard[0];
|
||||
for (const type of clipboardItem.types) {
|
||||
if (allowedLinkTypes.has(type)) {
|
||||
const blob = await clipboardItem.getType(type);
|
||||
const blobText = await blob.text();
|
||||
|
||||
return blobText;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -69,6 +69,15 @@
|
|||
/>
|
||||
</SettingsCategory>
|
||||
|
||||
<SettingsCategory sectionId="h265" title={$t("settings.video.h265")}>
|
||||
<SettingsToggle
|
||||
settingContext="save"
|
||||
settingId="tiktokH265"
|
||||
title={$t("settings.video.h265.title")}
|
||||
description={$t("settings.video.h265.description")}
|
||||
/>
|
||||
</SettingsCategory>
|
||||
|
||||
<SettingsCategory sectionId="twitter" title={$t("settings.video.twitter.gif")}>
|
||||
<SettingsToggle
|
||||
settingContext="save"
|
||||
|
|
@ -78,11 +87,3 @@
|
|||
/>
|
||||
</SettingsCategory>
|
||||
|
||||
<SettingsCategory sectionId="tiktok" title={$t("settings.video.tiktok.h265")}>
|
||||
<SettingsToggle
|
||||
settingContext="save"
|
||||
settingId="tiktokH265"
|
||||
title={$t("settings.video.tiktok.h265.title")}
|
||||
description={$t("settings.video.tiktok.h265.description")}
|
||||
/>
|
||||
</SettingsCategory>
|
||||
|
|
|
|||
Loading…
Reference in a new issue