Compare commits

...

14 Commits

Author SHA1 Message Date
GHStaK b9001b7aac Document gemini rollback baseline
build-push / docker (push) Successful in 4m39s
2026-03-18 13:03:02 +09:00
GHStaK 40a2f817fd Revert "Harden gemini vision JSON recovery"
This reverts commit 513199f426.
2026-03-18 13:00:41 +09:00
GHStaK 9a33ecc6b5 Revert "Reduce gemini partial batch noise"
This reverts commit 3be797131a.
2026-03-18 13:00:41 +09:00
GHStaK 770aea0f57 Revert "Harden single-candidate gemini recovery"
This reverts commit b6a217cab9.
2026-03-18 13:00:40 +09:00
GHStaK acfad750ab Revert "Replace gemini batch JSON protocol"
This reverts commit f5d76fc3ec.
2026-03-18 13:00:40 +09:00
GHStaK f5d76fc3ec Replace gemini batch JSON protocol
build-push / docker (push) Successful in 4m3s
2026-03-17 17:41:33 +09:00
GHStaK b6a217cab9 Harden single-candidate gemini recovery
build-push / docker (push) Successful in 4m14s
2026-03-17 17:23:05 +09:00
GHStaK 3be797131a Reduce gemini partial batch noise
build-push / docker (push) Successful in 4m9s
2026-03-17 17:00:51 +09:00
GHStaK 513199f426 Harden gemini vision JSON recovery
build-push / docker (push) Successful in 4m13s
2026-03-17 16:33:09 +09:00
GHStaK 91ee37593c Fix gemini candidate starvation
build-push / docker (push) Successful in 4m7s
2026-03-17 16:06:59 +09:00
GHStaK 139e8f8781 Add Windows PowerShell dev workflow
build-push / docker (push) Successful in 4m7s
2026-03-17 15:47:01 +09:00
AI Assistant 75f1bb360c Reduce noisy Gemini partial failure counts
build-push / docker (push) Successful in 4m8s
2026-03-17 15:09:23 +09:00
AI Assistant 58d54a0338 Fix 504 after restoring broader search baseline
build-push / docker (push) Successful in 4m21s
2026-03-17 14:07:33 +09:00
AI Assistant c177bae59e Reapply "Strengthen search breadth and modal fitting"
This reverts commit 3f824c4bdf.
2026-03-17 14:04:07 +09:00
18 changed files with 896 additions and 70 deletions
+4
View File
@@ -6,3 +6,7 @@ worker/__pycache__/
*.pyc
node_modules/
dist/
.venv/
.tools/
.local/
*.log
+152 -13
View File
@@ -8,6 +8,11 @@
- how it was verified
- what is still risky or incomplete
- If a push fails or a change remains local-only, that must be written here explicitly.
- Active planning and handoff notes should be written in Korean.
- After each meaningful completed task:
- update this file in context
- push the latest git state when operationally possible
- Local git credentials used for push automation must stay in an ignored local-only file and must never be committed.
## Current State At A Glance
- Project: `ai-media-hub`
@@ -33,7 +38,9 @@
- Result modal sizing is now being constrained to the viewport, and modal-only source-summary translation is now part of the active implementation path.
- Card summaries now also translate lazily to Korean, and Gemini negative-assessment handling now drives stronger follow-up search behavior than before.
- Search preview delivery is now moving away from persistent on-disk preview caching toward live proxy / live transcode behavior, with Google Video preview reuse added to result cards and modal playback.
- The latest search-breadth / modal-fitting experiment from `5ca7aef` has been rolled back after live regression was confirmed.
- Search breadth and Gemini review budget were widened again because the latest user feedback still reported a thin visible result count even after smarter filtering.
- The codebase is now back on the broader-search / modal-fitting direction associated with `5ca7aef`, with an added Gemini deadline guard to reduce reverse-proxy `504` risk.
- Windows 11 local development now has a repo-local PowerShell bootstrap / run / self-test / push workflow built around `.venv`, `.tools`, and `.local` so the machine does not need global Go / ffmpeg / yt-dlp changes for this repo.
## Current Architecture
- `backend/main.go`
@@ -216,6 +223,7 @@
- [x] Local self-test workflow
- [x] Source-specific search collectors
- [x] Shared ranker service layer
- [x] Windows 11 PowerShell local bootstrap / self-test / push workflow
## Important Current Constraints / Known Problems
- Search backend quality is still the most fragile subsystem.
@@ -232,7 +240,9 @@
- Source Summary translation now depends on Google Translate HTTP availability; frontend silently falls back to original summary text if translation fails.
- The result modal should now stay within viewport height, but this still needs real browser confirmation on multiple short-height displays because CSS-only constraints were the source of the latest user-visible regression.
- Artgrid preview playback now has a server-side ffmpeg transcode path for `.m3u8` style preview URLs, but this trades storage savings for runtime CPU cost.
- The reverted `5ca7aef` experiment showed that simply widening collector caps and Gemini candidate count can backfire when the added candidates are weak: final visible count fell sharply even though backend raw candidate count increased.
- The provided Artgrid HTML sample still does not expose a direct preview `m3u8` or `mp4` URL by itself, and `yt-dlp` probe on the sample clip URL returned `Unsupported URL`, so fully reliable Artgrid playback still depends on live-page/runtime preview discovery succeeding elsewhere in the pipeline.
- Docker CLI is not installed in this environment, so container-image build verification still cannot be performed locally from this machine.
- Gemini batch warnings can still mix true model/output failures with harmless candidate-skip cases unless the recovery path filters those error classes before surfacing them.
- The local self-test script is better than before, but it is still a smoke test, not full integration coverage.
## Current Risks Around Search Quality
@@ -423,6 +433,24 @@
- `node` is still not installed on this machine.
- This is acceptable for the current repo because there is still no Node-based frontend build or lint workflow in-tree.
- If future frontend work adds a Node toolchain, document the exact version and setup steps here before pushing.
- Windows 11 repo-local workflow:
- `scripts/setup-dev.ps1`
- creates `.venv`
- installs `worker/requirements.txt` into the repo-local venv
- downloads repo-local Go into `.tools/go`
- downloads repo-local ffmpeg into `.tools/ffmpeg`
- creates local command shims under `.tools/bin` such as `python3.cmd`
- `scripts/enter-dev-shell.ps1`
- prepends `.venv`, `.tools/bin`, local Go, and local ffmpeg to `PATH`
- pins `GOPATH`, `GOMODCACHE`, and `GOCACHE` into `.local` so Go does not need user-profile write access
- `scripts/run-dev.ps1`
- launches the backend with Windows-friendly local environment wiring
- `scripts/selftest.ps1`
- PowerShell-native replacement for the Linux shell smoke test
- verifies `gofmt`, Python syntax, `go test ./...`, frontend syntax via `node --check`, backend build, mock SearXNG boot, `/healthz`, `/api/search`, and `/api/upload`
- `scripts/push.ps1`
- reads ignored local credentials from `.local/git-credentials.psd1`
- pushes without storing credentials in tracked files
## Local Self-Test Workflow
- Primary command:
@@ -554,11 +582,14 @@
## Highest-Value Next Steps
- [ ] Reduce `/api/search` latency further without collapsing result count
- [ ] Rebuild the reverted search-expansion work from the previous stable baseline, but only after measuring where candidate quality collapses between ranked pool and final merge
- [ ] Validate the reopened `5ca7aef` search-breadth direction against real proxy timeouts and visible result count before widening it any further
- [ ] Continue hardening Gemini batch parsing so truncated JSON and ignorable low-visual candidate failures do not inflate user-facing warning counts
- [ ] Build a repeatable repo-local bootstrap script or documented setup command set for non-root machines so fresh PC setup does not depend on shell history
- [ ] Improve Envato / Artgrid preview acquisition reliability so Gemini Vision sees real frames more often
- [ ] Browser-verify the new result modal at multiple viewport heights and confirm translated Source Summary readability on real long descriptions
- [ ] Evaluate whether the new Gemini supplemental-query generation is reducing irrelevant results on a small fixed benchmark query set
- [ ] Measure runtime cost of live Artgrid preview transcoding and decide whether bounded in-memory throttling or concurrency caps are needed
- [ ] If Artgrid playback is still required for previewless clips, investigate a browser-rendered fetch path or clip-page asset bundle analysis because static HTML + `yt-dlp` probe were both insufficient on the supplied sample
- [ ] Revisit Google Video UX:
- current YouTube embed was abandoned due error `153`
- current in-app panel is more reliable but less rich than a true embedded watch page
@@ -624,24 +655,132 @@
- If behavior in the browser does not match the latest backend/frontend code, the first assumption should be stale frontend assets until proven otherwise
## Recent Change Log
- Date: `2026-03-18`
- What changed:
- Reverted the recent Gemini output-format experimentation commits so the repo returns to the last simpler known-good search/Gemini state anchored by `91ee375`.
- Reverted commits:
- `f5d76fc` `Replace gemini batch JSON protocol`
- `b6a217c` `Harden single-candidate gemini recovery`
- `3be7971` `Reduce gemini partial batch noise`
- `513199f` `Harden gemini vision JSON recovery`
- The effective runtime baseline after the rollback is now the `91ee375` state plus the newer Windows 11 PowerShell workflow work.
- Why it changed:
- Repeated Gemini output-format changes were still producing real user-facing failures across multiple logs, including:
- `gemini vision failed for all batches`
- `gemini vision partially failed on 4 of 6 batches`
- extremely short truncated payloads such as `"{\"recommend"` and `"{\"recommendations\":[{\"index"`
- The user explicitly requested a rollback to a commit state that worked normally instead of continuing to stack more Gemini parsing experiments.
- How it was verified:
- `pwsh -NoProfile -File scripts/selftest.ps1`
- local git history inspection confirmed the rollback point as the pre-experiment Gemini baseline `91ee375`
- What is still risky or incomplete:
- This rollback intentionally gives up the later Gemini parsing experiments, so the codebase no longer contains those attempted mitigations.
- If the original upstream Gemini truncation issue already existed before those follow-up commits, it can still reappear and would need a cleaner redesign from the reverted baseline rather than more incremental patching on top of the discarded branch.
- Date: `2026-03-17`
- What changed:
- Reverted commit `5ca7aef` (`Strengthen search breadth and modal fitting`) to restore the previous stable search/modal baseline.
- Revalidated the rollback state locally.
- Fixed a search-budget regression where source collection could consume the full `SearchService` deadline and leave no time for Envato / Artgrid enrichment, causing Gemini to see only missing or low-value visuals.
- Split the search-service deadline into:
- collector deadline
- enrichment deadline with an explicit reserved window
- Added unit coverage for the new deadline split behavior.
- Stopped frontend preview-probe fallback from calling `/api/download/preview` for Artgrid items that do not already have a provider preview URL, so unsupported `yt-dlp` Artgrid probe errors no longer fire just from opening or hovering those results.
- Why it changed:
- The user reported that search results became too sparse again and that `search returned partial results to avoid gateway timeout` reappeared.
- The provided log `ai-media-hub-2026-03-17T04-19-08-889Z.log` showed:
- backend candidate pool still reaching `30`
- Gemini candidate cap widened to `24`
- `visualRejectCount: 12`
- final visible result count collapsing to `5`
- That indicates the widened experiment increased weak/low-visual candidates and pushed the request back toward deadline pressure without improving visible result count.
- The user-provided log `ai-media-hub-2026-03-17T07-01-21-282Z.log` showed:
- `search_service:deadline_reached`
- immediate `search_service:enrich_start` -> `search_service:enrich_complete`
- `withPreview: 0`
- `withLowValueThumbnail: 12`
- repeated `candidate has no thumbnail or preview video`
- final warning `gemini vision returned no candidate evaluations`
- The same log also showed Artgrid preview probe failures from `yt-dlp` returning `Unsupported URL`, which were not helping user-facing preview behavior.
- How it was verified:
- `pwsh -NoProfile -File scripts/selftest.ps1`
- added Go tests for the search/enrichment deadline split helper
- What is still risky or incomplete:
- This preserves time for enrichment, but it does not guarantee that every live Envato / Artgrid page yields a usable preview URL.
- Artgrid still depends on backend-enriched provider preview URLs for true video preview; if no provider preview is discovered, the UI will still fall back to thumbnail-only rendering.
- Date: `2026-03-17`
- What changed:
- Added repo-local Windows 11 PowerShell workflows:
- `scripts/dev-tools.ps1`
- `scripts/setup-dev.ps1`
- `scripts/enter-dev-shell.ps1`
- `scripts/run-dev.ps1`
- `scripts/selftest.ps1`
- `scripts/push.ps1`
- Added ignored local directories for `.venv`, `.tools`, and `.local`.
- Created a local-only git credential file for automated push flow and kept it excluded from git.
- Pinned PowerShell tooling to repo-local Go caches under `.local` so `go test` no longer depends on writable user-profile Go paths.
- Why it changed:
- The active development machine is now Windows 11, and the user requested that setup, verification, and push flows work through PowerShell while minimizing machine-wide side effects.
- The previous local workflow was Linux-shell-oriented and did not directly cover Windows PowerShell usage.
- Initial Windows self-test attempts failed because Go wanted to write into the default user Go cache path, which was not reliably writable in this environment.
- How it was verified:
- `pwsh -NoProfile -File scripts/setup-dev.ps1 -SkipPythonDeps -SkipGoDownload -SkipFFmpegDownload`
- elevated `pwsh -NoProfile -File scripts/setup-dev.ps1`
- `pwsh -NoProfile -Command ". .\scripts\enter-dev-shell.ps1; python3 --version; yt-dlp --version; go version; ffmpeg -version | Select-Object -First 1"`
- `pwsh -NoProfile -File scripts/selftest.ps1`
- `git check-ignore -v .local\git-credentials.psd1 .venv .tools`
- What is still risky or incomplete:
- `scripts/setup-dev.ps1` still requires network access for Python package install and local Go / ffmpeg downloads on a truly fresh machine.
- The backend runtime still invokes `python3` by name, so Windows usage depends on entering the repo-local dev shell or otherwise exposing the generated shim on `PATH`.
- Push automation now uses an ignored local credential file, but credential rotation remains a manual step.
- Date: `2026-03-17`
- What changed:
- Restored hard/ignorable Gemini batch error filtering so low-value thumbnail skips and no-visual candidate skips do not count as user-facing partial batch failures when useful recommendations were still recovered.
- Added unit coverage for the Gemini batch error filtering behavior.
- Why it changed:
- The user reported the warning `gemini vision partially failed on 3 of 4 batches`.
- The provided log `ai-media-hub-2026-03-17T06-06-04-447Z.log` showed three different batch-failure classes mixed together:
- real Gemini JSON extraction failures
- low-value thumbnail skips
- no-visual candidate skips
- Only the first class should strongly influence the user-facing partial-failure warning.
- How it was verified:
- `go test ./...`
- `bash scripts/selftest.sh`
- What is still risky or incomplete:
- This rollback restores the earlier baseline but does not solve the underlying request to improve visible result count.
- The next attempt should start from the reverted baseline and target the actual bottleneck between ranked pool and final merge, instead of only widening raw query breadth.
- This reduces noisy partial-failure warnings, but it does not eliminate genuine Gemini JSON truncation or provider-side preview problems.
- If the model starts returning different malformed JSON patterns, the parser and warning logic may still need further hardening.
- Date: `2026-03-17`
- What changed:
- Reapplied the broader-search / modal-fitting codepath from `5ca7aef` as requested by the user.
- Added a targeted 504 mitigation by making Gemini batch recovery stop at the request deadline instead of continuing sequential single-candidate retries indefinitely.
- Kept an explicit Gemini vision JSON output token cap to reduce the chance of truncated model responses during larger structured batches.
- Why it changed:
- The user explicitly asked to return to the `5ca7aef` direction and then fix the live `504 Gateway Time-out`.
- The provided log `ai-media-hub-2026-03-17T04-59-24-566Z.log` showed backend search collection finishing in about `22s`, then Gemini batch failure plus sequential recovery continuing until the reverse proxy returned `504`.
- The direct cause was deadline-unaware Gemini recovery work, not initial search collection itself.
- How it was verified:
- `go test ./...`
- `bash scripts/selftest.sh`
- attempted `docker build -t ai-media-hub:test .` but `docker` is not installed in this environment
- What is still risky or incomplete:
- This reduces one concrete `504` path, but it does not guarantee that all reverse-proxy timeout cases are eliminated under worse upstream conditions.
- Container-image build verification still needs to happen on a machine that actually has Docker available.
- Date: `2026-03-17`
- What changed:
- Raised search breadth again by widening per-source collector caps, increasing the number of base queries considered, increasing per-collector query budgets, and expanding the Gemini candidate review budget.
- Added another browser-resolution-aware modal fitting pass: the result popup now recalculates shell width, shell height, media max height, and compact-mode behavior from `window.innerWidth` / `window.innerHeight` when opened or when the viewport changes.
- Removed the fixed minimum heights on the lower modal panels so they can shrink with the viewport instead of forcing the popup past the visible browser area.
- Checked the supplied Artgrid clip HTML and a direct `yt-dlp` probe against `https://artgrid.io/clip/355470/...`; the HTML still exposes metadata and thumbnail signals only, and the direct probe returned `Unsupported URL`.
- Why it changed:
- The user reported that the popup still overflowed the browser viewport and that the visible result set still felt too thin despite the earlier search-expansion work.
- The attached Artgrid HTML sample was intended to verify whether a stable preview URL could be extracted directly from static clip HTML.
- How it was verified:
- `go test ./...`
- `bash scripts/selftest.sh`
- `python3 -m py_compile worker/downloader.py scripts/mock_searxng.py`
- `python3 worker/downloader.py --mode probe --url 'https://artgrid.io/clip/355470/couple-woman-man-holding-hand'`
- What is still risky or incomplete:
- The new modal layout now adapts to actual browser viewport dimensions in code, but a live browser confirmation on the exact user display is still needed.
- Widening search breadth and Gemini review budget should improve visible count, but it can also increase latency and token cost under poor upstream conditions.
- The supplied Artgrid HTML sample alone was not enough to derive a directly playable preview URL, so previewless Artgrid clips may still fall back to thumbnail-only behavior.
- Date: `2026-03-17`
- What changed:
+27 -24
View File
@@ -554,7 +554,7 @@ func (a *App) searchMedia(c *gin.Context) {
return
}
targetCount := 16
targetCount := 18
merged := services.MergeRecommendations(recommended, scored, targetCount)
if geminiErr != nil {
merged = services.BackfillRecommendations(
@@ -564,31 +564,34 @@ func (a *App) searchMedia(c *gin.Context) {
"Gemini 배치 일부가 실패해 미리보기 가능한 상위 후보를 보강했습니다.",
)
}
if len(merged) < targetCount && time.Now().Before(deadline.Add(-5*time.Second)) {
for pass := 0; pass < 2 && len(merged) < targetCount && time.Now().Before(deadline.Add(-4*time.Second)); pass++ {
coverageQueries := buildCoverageQueries(req.Query, queryVariants, recommended, merged)
if len(coverageQueries) > 0 {
a.debug("search coverage query variants", gin.H{"variants": coverageQueries, "variantCount": len(coverageQueries), "existingCount": len(merged)})
extraResults, extraMeta, extraErr := a.SearchService.SearchMediaWithDeadline(coverageQueries, enabledPlatforms, deadline.Add(-5*time.Second))
supplementalDeadlineLimited = supplementalDeadlineLimited || extraMeta.PartialDueToDeadline
if extraErr == nil && len(extraResults) > 0 {
results = mergeSearchResults(results, extraResults)
scored = services.RankSearchResults(strings.Join(coverageQueries[:min(len(coverageQueries), 3)], " "), results)
reviewedLinks := services.ReviewedRecommendationLinks(recommended)
supplementalCandidates := services.SelectUnevaluatedCandidates(scored, reviewedLinks, services.RemainingGeminiCapacity(recommended))
if len(supplementalCandidates) > 0 {
extraRecommended, extraStats, extraGeminiErr := services.EvaluateAllCandidatesWithGeminiWithDeadline(
a.GeminiService,
req.Query,
supplementalCandidates,
deadline.Add(-2*time.Second),
)
recommended = services.MergeUniqueRecommendations(recommended, extraRecommended)
geminiStats = services.MergeGeminiBatchStats(geminiStats, extraStats)
geminiErr = combineSearchWarnings(geminiErr, extraGeminiErr)
}
merged = services.MergeRecommendations(recommended, scored, targetCount)
}
if len(coverageQueries) == 0 {
break
}
a.debug("search coverage query variants", gin.H{"variants": coverageQueries, "variantCount": len(coverageQueries), "existingCount": len(merged), "pass": pass + 1})
extraResults, extraMeta, extraErr := a.SearchService.SearchMediaWithDeadline(coverageQueries, enabledPlatforms, deadline.Add(-4*time.Second))
supplementalDeadlineLimited = supplementalDeadlineLimited || extraMeta.PartialDueToDeadline
if extraErr != nil || len(extraResults) == 0 {
break
}
results = mergeSearchResults(results, extraResults)
scored = services.RankSearchResults(strings.Join(coverageQueries[:min(len(coverageQueries), 3)], " "), results)
reviewedLinks := services.ReviewedRecommendationLinks(recommended)
supplementalCandidates := services.SelectUnevaluatedCandidates(scored, reviewedLinks, services.RemainingGeminiCapacity(recommended))
if len(supplementalCandidates) == 0 {
break
}
extraRecommended, extraStats, extraGeminiErr := services.EvaluateAllCandidatesWithGeminiWithDeadline(
a.GeminiService,
req.Query,
supplementalCandidates,
deadline.Add(-2*time.Second),
)
recommended = services.MergeUniqueRecommendations(recommended, extraRecommended)
geminiStats = services.MergeGeminiBatchStats(geminiStats, extraStats)
geminiErr = combineSearchWarnings(geminiErr, extraGeminiErr)
merged = services.MergeRecommendations(recommended, scored, targetCount)
}
if len(merged) < targetCount {
merged = services.BackfillRecommendations(merged, scored, targetCount, "추가 검색 후에도 충분한 결과가 부족해 시각 자산이 있는 후보를 제한적으로 보강했습니다.")
+23 -8
View File
@@ -53,6 +53,8 @@ type SearchExecutionMeta struct {
PartialDueToDeadline bool `json:"partialDueToDeadline"`
}
const searchEnrichmentReserve = 4 * time.Second
func NewSearchService(baseURL, googleVideoEngine, webEngine string) *SearchService {
if googleVideoEngine == "" {
googleVideoEngine = "google videos"
@@ -84,9 +86,11 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
if s.BaseURL == "" {
return nil, meta, fmt.Errorf("searxng base url is not configured")
}
collectionDeadline, enrichmentDeadline := splitSearchDeadlines(deadline)
s.debug("search_service:start", map[string]any{
"queries": queries,
"enabledPlatforms": enabledPlatforms,
"deadlineSet": !deadline.IsZero(),
})
seen := map[string]bool{}
@@ -94,12 +98,12 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
results := make([]SearchResult, 0, 90)
var lastErr error
baseQueries := limitQueries(queries, 8)
baseQueries := limitQueries(queries, 10)
shuffleStrings(baseQueries)
primaryQueries := baseQueries[:minInt(len(baseQueries), 3)]
primaryQueries := baseQueries[:minInt(len(baseQueries), 4)]
runSearchPass := func(bases []string, onlyMissing bool) {
for _, base := range bases {
if !deadline.IsZero() && time.Now().After(deadline) {
if !collectionDeadline.IsZero() && time.Now().After(collectionDeadline) {
meta.PartialDueToDeadline = true
s.debug("search_service:deadline_reached", map[string]any{"stage": "runSearchPass", "base": base})
return
@@ -109,7 +113,7 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
continue
}
for _, collector := range s.collectors {
if !deadline.IsZero() && time.Now().After(deadline) {
if !collectionDeadline.IsZero() && time.Now().After(collectionDeadline) {
meta.PartialDueToDeadline = true
s.debug("search_service:deadline_reached", map[string]any{"stage": "collectorLoop", "collector": collector.Name()})
return
@@ -133,7 +137,7 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
"searchQueries": searchQueries,
})
for _, searchQuery := range searchQueries {
if !deadline.IsZero() && time.Now().After(deadline) {
if !collectionDeadline.IsZero() && time.Now().After(collectionDeadline) {
meta.PartialDueToDeadline = true
s.debug("search_service:deadline_reached", map[string]any{"stage": "queryLoop", "collector": collector.Name(), "query": searchQuery})
return
@@ -192,11 +196,22 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
"hadError": lastErr != nil,
"partialDueToDeadline": meta.PartialDueToDeadline,
})
enriched, enrichMeta := s.EnrichResultsWithDeadline(results, deadline)
enriched, enrichMeta := s.EnrichResultsWithDeadline(results, enrichmentDeadline)
meta.PartialDueToDeadline = meta.PartialDueToDeadline || enrichMeta.PartialDueToDeadline
return enriched, meta, nil
}
func splitSearchDeadlines(deadline time.Time) (time.Time, time.Time) {
if deadline.IsZero() {
return time.Time{}, time.Time{}
}
remaining := time.Until(deadline)
if remaining <= searchEnrichmentReserve {
return deadline, deadline
}
return deadline.Add(-searchEnrichmentReserve), deadline
}
func (s *SearchService) EnrichResults(results []SearchResult) []SearchResult {
enriched, _ := s.EnrichResultsWithDeadline(results, time.Time{})
return enriched
@@ -1432,9 +1447,9 @@ func limitCollectorQueries(collector string, queries []string, onlyMissing bool)
limit := 2
switch collector {
case "Envato", "Artgrid":
limit = 4
limit = 5
case "Google Video":
limit = 3
limit = 4
}
if onlyMissing {
limit--
+28 -4
View File
@@ -159,13 +159,13 @@ func TestLimitCollectorQueriesUsesSmallerBudgetForMissingPass(t *testing.T) {
queries := []string{"a", "b", "c", "d"}
got := limitCollectorQueries("Artgrid", queries, true)
if len(got) != 3 {
t.Fatalf("expected 3 queries for missing-pass Artgrid collector, got %d", len(got))
if len(got) != 4 {
t.Fatalf("expected 4 queries for missing-pass Artgrid collector, got %d", len(got))
}
got = limitCollectorQueries("Google Video", queries, false)
if len(got) != 3 {
t.Fatalf("expected 3 queries for Google Video collector, got %d", len(got))
if len(got) != 4 {
t.Fatalf("expected 4 queries for Google Video collector, got %d", len(got))
}
}
@@ -182,6 +182,30 @@ func TestSearchServiceFetchCacheRoundTrip(t *testing.T) {
}
}
func TestSplitSearchDeadlinesReservesEnrichmentWindow(t *testing.T) {
deadline := time.Now().Add(20 * time.Second)
collectionDeadline, enrichmentDeadline := splitSearchDeadlines(deadline)
if enrichmentDeadline.IsZero() {
t.Fatal("expected enrichment deadline to be preserved")
}
if !collectionDeadline.Before(enrichmentDeadline) {
t.Fatalf("expected collection deadline before enrichment deadline, got %v >= %v", collectionDeadline, enrichmentDeadline)
}
if gap := enrichmentDeadline.Sub(collectionDeadline); gap < searchEnrichmentReserve-500*time.Millisecond {
t.Fatalf("expected reserve close to %v, got %v", searchEnrichmentReserve, gap)
}
}
func TestSplitSearchDeadlinesDoesNotReserveWhenDeadlineIsTooClose(t *testing.T) {
deadline := time.Now().Add(2 * time.Second)
collectionDeadline, enrichmentDeadline := splitSearchDeadlines(deadline)
if !collectionDeadline.Equal(enrichmentDeadline) {
t.Fatalf("expected identical deadlines when budget is too tight, got %v and %v", collectionDeadline, enrichmentDeadline)
}
}
func TestSearchServiceSkipsArtgridAPIAfter403(t *testing.T) {
var apiRequests atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+1
View File
@@ -306,6 +306,7 @@ User query: ` + query,
},
"generationConfig": map[string]any{
"responseMimeType": "application/json",
"maxOutputTokens": 1400,
},
}
+17 -2
View File
@@ -123,8 +123,8 @@ func TestRemainingGeminiCapacityShrinksWithReviewedItems(t *testing.T) {
{Link: "https://a.example"},
{Link: "https://b.example"},
}
if got := RemainingGeminiCapacity(reviewed); got != 14 {
t.Fatalf("expected 14 remaining slots, got %d", got)
if got := RemainingGeminiCapacity(reviewed); got != 22 {
t.Fatalf("expected 22 remaining slots, got %d", got)
}
}
@@ -213,3 +213,18 @@ func TestMergeRecommendationsExcludesIrrelevantAndPendingFiller(t *testing.T) {
t.Fatalf("unexpected merged result: %#v", merged)
}
}
func TestFilterHardGeminiErrorsIgnoresLowValueVisualFailures(t *testing.T) {
errs := []string{
"candidate thumbnail is low value",
"no candidate thumbnails or preview frames could be fetched for gemini vision",
"gemini vision JSON extraction failed: no complete JSON object found",
}
filtered := filterHardGeminiErrors(errs)
if len(filtered) != 1 {
t.Fatalf("expected only hard errors to remain, got %#v", filtered)
}
if !strings.Contains(filtered[0], "JSON extraction failed") {
t.Fatalf("unexpected filtered errors: %#v", filtered)
}
}
+48 -9
View File
@@ -97,7 +97,7 @@ func RankSearchResults(query string, results []SearchResult) []SearchResult {
}
func GeminiCandidateLimit(total int) int {
return min(total, 16)
return min(total, 24)
}
func EvaluateAllCandidatesWithGemini(service *GeminiService, query string, ranked []SearchResult) ([]AIRecommendation, GeminiBatchStats, error) {
@@ -105,7 +105,7 @@ func EvaluateAllCandidatesWithGemini(service *GeminiService, query string, ranke
}
func EvaluateAllCandidatesWithGeminiWithDeadline(service *GeminiService, query string, ranked []SearchResult, deadline time.Time) ([]AIRecommendation, GeminiBatchStats, error) {
const chunkSize = 8
const chunkSize = 6
const maxConcurrentBatches = 2
if service == nil {
return nil, GeminiBatchStats{}, fmt.Errorf("gemini service is not configured")
@@ -186,7 +186,8 @@ func EvaluateAllCandidatesWithGeminiWithDeadline(service *GeminiService, query s
"error": batch.err.Error(),
})
}
recovered, recoveredErrs := recoverGeminiBatchSequentially(service, query, ranked, batch.index*chunkSize)
recovered, recoveredErrs := recoverGeminiBatchSequentially(service, query, ranked, batch.index*chunkSize, chunkSize, deadline)
hardErrs := filterHardGeminiErrors(recoveredErrs)
if len(recovered) > 0 {
stats.SequentialRetried++
stats.Succeeded++
@@ -197,9 +198,9 @@ func EvaluateAllCandidatesWithGeminiWithDeadline(service *GeminiService, query s
seen[item.Link] = true
merged = append(merged, item)
}
if len(recoveredErrs) > 0 {
if len(hardErrs) > 0 {
stats.Failed++
for _, recoveredErr := range recoveredErrs {
for _, recoveredErr := range hardErrs {
if len(stats.Errors) < 5 {
stats.Errors = append(stats.Errors, recoveredErr)
}
@@ -207,6 +208,9 @@ func EvaluateAllCandidatesWithGeminiWithDeadline(service *GeminiService, query s
}
continue
}
if len(hardErrs) == 0 {
continue
}
stats.Failed++
if len(stats.Errors) < 5 {
stats.Errors = append(stats.Errors, batch.err.Error())
@@ -291,7 +295,7 @@ func ReviewedRecommendationLinks(items []AIRecommendation) map[string]bool {
}
func RemainingGeminiCapacity(reviewed []AIRecommendation) int {
remaining := GeminiCandidateLimit(16) - len(ReviewedRecommendationLinks(reviewed))
remaining := GeminiCandidateLimit(24) - len(ReviewedRecommendationLinks(reviewed))
if remaining < 0 {
return 0
}
@@ -345,11 +349,46 @@ func MergeGeminiBatchStats(base, extra GeminiBatchStats) GeminiBatchStats {
return merged
}
func recoverGeminiBatchSequentially(service *GeminiService, query string, ranked []SearchResult, startIndex int) ([]AIRecommendation, []string) {
recovered := make([]AIRecommendation, 0, 8)
func filterHardGeminiErrors(errs []string) []string {
filtered := make([]string, 0, len(errs))
for _, item := range errs {
if isIgnorableGeminiError(item) {
continue
}
filtered = append(filtered, item)
}
return filtered
}
func isIgnorableGeminiError(message string) bool {
lower := strings.ToLower(strings.TrimSpace(message))
if lower == "" {
return false
}
for _, token := range []string{
"no candidate thumbnails or preview frames could be fetched for gemini vision",
"candidate thumbnail is low value",
"candidate has no thumbnail or preview video",
"image url is empty",
} {
if strings.Contains(lower, token) {
return true
}
}
return false
}
func recoverGeminiBatchSequentially(service *GeminiService, query string, ranked []SearchResult, startIndex, chunkSize int, deadline time.Time) ([]AIRecommendation, []string) {
recovered := make([]AIRecommendation, 0, chunkSize)
errs := make([]string, 0, 4)
endIndex := min(startIndex+8, len(ranked))
endIndex := min(startIndex+chunkSize, len(ranked))
for idx := startIndex; idx < endIndex; idx++ {
if !deadline.IsZero() && time.Now().After(deadline) {
if len(errs) < 4 {
errs = append(errs, "sequential gemini recovery stopped at deadline")
}
break
}
recs, err := service.Recommend(query, []SearchResult{ranked[idx]})
if err != nil {
if len(errs) < 4 {
+3 -3
View File
@@ -15,7 +15,7 @@ type searchCollector interface {
type envatoCollector struct{}
func (envatoCollector) Name() string { return "Envato" }
func (envatoCollector) MaxResults() int { return 12 }
func (envatoCollector) MaxResults() int { return 16 }
func (envatoCollector) Enabled(enabledPlatforms map[string]bool) bool {
return len(enabledPlatforms) == 0 || enabledPlatforms["envato"]
}
@@ -31,7 +31,7 @@ func (envatoCollector) Enrich(searcher *SearchService, result SearchResult) Sear
type artgridCollector struct{}
func (artgridCollector) Name() string { return "Artgrid" }
func (artgridCollector) MaxResults() int { return 12 }
func (artgridCollector) MaxResults() int { return 16 }
func (artgridCollector) Enabled(enabledPlatforms map[string]bool) bool {
return len(enabledPlatforms) == 0 || enabledPlatforms["artgrid"]
}
@@ -47,7 +47,7 @@ func (artgridCollector) Enrich(searcher *SearchService, result SearchResult) Sea
type googleVideoCollector struct{}
func (googleVideoCollector) Name() string { return "Google Video" }
func (googleVideoCollector) MaxResults() int { return 8 }
func (googleVideoCollector) MaxResults() int { return 12 }
func (googleVideoCollector) Enabled(enabledPlatforms map[string]bool) bool {
return len(enabledPlatforms) == 0 || enabledPlatforms["google video"]
}
+26 -1
View File
@@ -38,6 +38,7 @@ const downloadLogs = document.getElementById("downloadLogs");
const debugLogList = document.getElementById("debugLogList");
const debugSummary = document.getElementById("debugSummary");
const resultModal = document.getElementById("resultModal");
const resultModalShell = document.getElementById("resultModalShell");
const resultModalTitle = document.getElementById("resultModalTitle");
const resultModalSource = document.getElementById("resultModalSource");
const resultModalSnippet = document.getElementById("resultModalSnippet");
@@ -56,6 +57,7 @@ const resultModalSecondaryAction = document.getElementById("resultModalSecondary
const closeResultModal = document.getElementById("closeResultModal");
const resultModalReady = Boolean(
resultModal &&
resultModalShell &&
resultModalTitle &&
resultModalSource &&
resultModalSnippet &&
@@ -436,12 +438,30 @@ function resetPreviewPlayer() {
function showModal(element) {
setHidden(element, false);
if (element === resultModal) {
syncResultModalLayout();
}
}
function hideModal(element) {
setHidden(element, true);
}
function syncResultModalLayout() {
if (!resultModalReady) {
return;
}
const viewportWidth = Math.max(320, window.innerWidth || document.documentElement.clientWidth || 0);
const viewportHeight = Math.max(320, window.innerHeight || document.documentElement.clientHeight || 0);
const shellWidth = Math.min(1150, viewportWidth - 16);
const shellHeight = Math.min(840, viewportHeight - 12);
const mediaHeight = Math.max(150, Math.min(Math.floor(viewportHeight * 0.32), 320));
resultModalShell.style.setProperty("--result-modal-shell-width", `${shellWidth}px`);
resultModalShell.style.setProperty("--result-modal-shell-height", `${shellHeight}px`);
resultModalShell.style.setProperty("--result-modal-media-max-height", `${mediaHeight}px`);
resultModalShell.classList.toggle("result-modal-compact", viewportHeight < 900 || viewportWidth < 1280);
}
function buildResultModalEmbedURL(item) {
if (item?.embedUrl) {
if (item.source === "Google Video" && item.embedUrl.includes("youtube-nocookie.com/embed/")) {
@@ -589,7 +609,7 @@ async function fetchResultPreview(item) {
}
const request = (async () => {
try {
const preview = await api("/api/download/preview", {
const preview = await api("/api/download/preview", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url: key }),
@@ -1030,6 +1050,11 @@ if (resultModalReady) {
}
});
}
window.addEventListener("resize", () => {
if (resultModalReady && !resultModal.classList.contains("hidden")) {
syncResultModalLayout();
}
});
for (const button of platformToggles) {
button.addEventListener("click", () => {
const platform = button.dataset.platformToggle;
+4 -4
View File
@@ -150,7 +150,7 @@
</div>
<div id="resultModal" class="fixed inset-0 z-50 hidden items-start justify-center overflow-y-auto bg-black/80 px-2 py-2 sm:px-4 sm:py-4">
<div class="result-modal-shell flex w-full max-w-6xl min-h-0 flex-col overflow-hidden rounded-3xl border border-white/10 bg-zinc-950 shadow-2xl">
<div id="resultModalShell" class="result-modal-shell flex w-full max-w-6xl min-h-0 flex-col overflow-hidden rounded-3xl border border-white/10 bg-zinc-950 shadow-2xl">
<div class="flex items-center justify-between border-b border-white/10 px-4 py-3 sm:px-5 sm:py-4">
<div class="min-w-0">
<p id="resultModalSource" class="text-xs uppercase tracking-[0.25em] text-zinc-500"></p>
@@ -180,13 +180,13 @@
</div>
</div>
<div class="result-modal-details grid min-h-0 gap-3 px-3 py-3 sm:gap-4 sm:px-4 sm:py-4 lg:grid-cols-[1.5fr_0.8fr]">
<div class="flex min-h-[180px] min-w-0 flex-col rounded-2xl border border-white/10 bg-white/[0.03] p-4">
<div class="flex min-h-0 min-w-0 flex-col rounded-2xl border border-white/10 bg-white/[0.03] p-4">
<p class="text-xs uppercase tracking-[0.25em] text-zinc-500">AI Note</p>
<div class="result-panel-scroll mt-3 min-h-0 flex-1 overflow-y-auto pr-2">
<p id="resultModalReason" class="whitespace-pre-wrap text-xs leading-6 text-zinc-200 sm:text-sm"></p>
</div>
</div>
<div class="flex min-h-[200px] min-w-0 flex-col rounded-2xl border border-white/10 bg-white/[0.03] p-4">
<div class="flex min-h-0 min-w-0 flex-col rounded-2xl border border-white/10 bg-white/[0.03] p-4">
<div class="mb-3 flex flex-col gap-2">
<button id="resultModalDownload" type="button" class="hidden w-full rounded-2xl border border-white bg-white px-4 py-3 text-sm font-medium text-black transition hover:bg-zinc-200">
Primary Action
@@ -224,6 +224,6 @@
</button>
</template>
<script src="/app.js?v=20260317c" defer></script>
<script src="/app.js?v=20260317d" defer></script>
</body>
</html>
+13 -2
View File
@@ -60,7 +60,8 @@ body {
}
.result-modal-shell {
height: min(calc(100dvh - 0.5rem), 860px);
height: var(--result-modal-shell-height, min(calc(100dvh - 0.5rem), 860px));
max-width: var(--result-modal-shell-width, 72rem);
margin: auto;
}
@@ -69,7 +70,7 @@ body {
}
.result-modal-media-frame {
max-height: min(34dvh, 22rem);
max-height: var(--result-modal-media-max-height, min(34dvh, 22rem));
}
.result-modal-details {
@@ -79,6 +80,16 @@ body {
align-items: stretch;
}
.result-modal-shell.result-modal-compact .result-modal-details {
gap: 0.5rem;
}
.result-modal-shell.result-modal-compact #resultModalReason,
.result-modal-shell.result-modal-compact #resultModalSnippet {
font-size: 0.75rem;
line-height: 1.35rem;
}
#resultModalSnippet,
#resultModalReason {
white-space: pre-wrap;
+280
View File
@@ -0,0 +1,280 @@
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$script:RepoRoot = Split-Path -Parent $PSScriptRoot
$script:ToolsRoot = Join-Path $script:RepoRoot ".tools"
$script:BinRoot = Join-Path $script:ToolsRoot "bin"
$script:GoRoot = Join-Path $script:ToolsRoot "go"
$script:GoBin = Join-Path $script:GoRoot "bin"
$script:FFmpegRoot = Join-Path $script:ToolsRoot "ffmpeg"
$script:FFmpegBin = Join-Path $script:FFmpegRoot "bin"
$script:VenvRoot = Join-Path $script:RepoRoot ".venv"
$script:VenvScripts = Join-Path $script:VenvRoot "Scripts"
$script:LocalRoot = Join-Path $script:RepoRoot ".local"
$script:GoPathRoot = Join-Path $script:LocalRoot "go"
$script:GoModCacheRoot = Join-Path $script:GoPathRoot "pkg\mod"
$script:GoBuildCacheRoot = Join-Path $script:LocalRoot "go-build-cache"
$script:CredentialFile = Join-Path $script:LocalRoot "git-credentials.psd1"
function Write-Step {
param([string]$Message)
Write-Host "[ai-media-hub] $Message"
}
function Invoke-CheckedCommand {
param(
[Parameter(Mandatory = $true)][string]$FilePath,
[string[]]$Arguments = @()
)
& $FilePath @Arguments
if ($LASTEXITCODE -ne 0) {
$joined = if ($Arguments.Count -gt 0) { " " + ($Arguments -join " ") } else { "" }
throw "명령 실행 실패: $FilePath$joined"
}
}
function Ensure-Directory {
param([Parameter(Mandatory = $true)][string]$Path)
if (-not (Test-Path -LiteralPath $Path)) {
New-Item -ItemType Directory -Path $Path | Out-Null
}
}
function Get-RepoRoot { return $script:RepoRoot }
function Get-ToolsRoot { return $script:ToolsRoot }
function Get-BinRoot { return $script:BinRoot }
function Get-GoRoot { return $script:GoRoot }
function Get-GoBin { return $script:GoBin }
function Get-FFmpegBin { return $script:FFmpegBin }
function Get-VenvRoot { return $script:VenvRoot }
function Get-VenvScripts { return $script:VenvScripts }
function Get-LocalRoot { return $script:LocalRoot }
function Get-GoPathRoot { return $script:GoPathRoot }
function Get-GoModCacheRoot { return $script:GoModCacheRoot }
function Get-GoBuildCacheRoot { return $script:GoBuildCacheRoot }
function Get-CredentialFile { return $script:CredentialFile }
function Resolve-SystemCommand {
param([Parameter(Mandatory = $true)][string]$Name)
try {
return (Get-Command $Name -ErrorAction Stop).Source
} catch {
return $null
}
}
function Resolve-PythonExe {
$candidates = @(
(Join-Path $script:VenvScripts "python.exe"),
(Resolve-SystemCommand -Name "python")
)
foreach ($candidate in $candidates) {
if ($candidate -and (Test-Path -LiteralPath $candidate)) {
return $candidate
}
}
throw "python.exe를 찾을 수 없습니다. Python 3.12+가 필요합니다."
}
function Resolve-BasePythonExe {
$all = @(Get-Command python -All -ErrorAction SilentlyContinue)
foreach ($item in $all) {
if ($item.Source -and $item.Source -notlike "*\.venv\*") {
return $item.Source
}
}
$fallback = Resolve-SystemCommand -Name "python"
if ($fallback -and $fallback -notlike "*\.venv\*") {
return $fallback
}
throw "시스템 python.exe를 찾지 못했습니다."
}
function Resolve-VenvPythonExe {
$pythonExe = Join-Path $script:VenvScripts "python.exe"
if (-not (Test-Path -LiteralPath $pythonExe)) {
throw ".venv가 아직 준비되지 않았습니다. scripts/setup-dev.ps1를 먼저 실행하세요."
}
return $pythonExe
}
function Use-LocalTooling {
Ensure-Directory -Path $script:ToolsRoot
Ensure-Directory -Path $script:BinRoot
Ensure-Directory -Path $script:LocalRoot
Ensure-Directory -Path $script:GoPathRoot
Ensure-Directory -Path $script:GoModCacheRoot
Ensure-Directory -Path $script:GoBuildCacheRoot
$segments = @(
$script:BinRoot,
$script:GoBin,
$script:FFmpegBin,
$script:VenvScripts,
$env:PATH
) | Where-Object { $_ -and $_.Trim() -ne "" }
$env:PATH = ($segments | Select-Object -Unique) -join ";"
$env:GOPATH = $script:GoPathRoot
$env:GOMODCACHE = $script:GoModCacheRoot
$env:GOCACHE = $script:GoBuildCacheRoot
}
function Ensure-Venv {
$venvPython = Join-Path $script:VenvScripts "python.exe"
if (Test-Path -LiteralPath $venvPython) {
return $venvPython
}
Ensure-Directory -Path $script:LocalRoot
$pythonExe = Resolve-SystemCommand -Name "python"
if (-not $pythonExe) {
throw "시스템 python이 없어 .venv를 만들 수 없습니다."
}
Write-Step ".venv 생성"
& $pythonExe -m venv $script:VenvRoot
return $venvPython
}
function Install-PythonRequirements {
param([switch]$UpgradePip)
$venvPython = Ensure-Venv
$systemPython = Resolve-BasePythonExe
if ($UpgradePip) {
Write-Step "pip 업그레이드"
Invoke-CheckedCommand -FilePath $systemPython -Arguments @("-m", "pip", "--python", $venvPython, "install", "--upgrade", "pip")
}
Write-Step "worker requirements 설치"
Invoke-CheckedCommand -FilePath $systemPython -Arguments @("-m", "pip", "--python", $venvPython, "install", "-r", (Join-Path $script:RepoRoot "worker\requirements.txt"))
}
function New-CmdShim {
param(
[Parameter(Mandatory = $true)][string]$Path,
[Parameter(Mandatory = $true)][string]$Target
)
$content = @(
"@echo off",
"set SCRIPT_DIR=%~dp0",
"`"%SCRIPT_DIR%$Target`" %*"
) -join "`r`n"
Set-Content -LiteralPath $Path -Value $content -Encoding ASCII
}
function Ensure-CommandShims {
Ensure-Directory -Path $script:BinRoot
$pythonShimTarget = "..\..\ .venv\Scripts\python.exe".Replace(" ", "")
New-CmdShim -Path (Join-Path $script:BinRoot "python3.cmd") -Target $pythonShimTarget
$ytDlpExe = Join-Path $script:VenvScripts "yt-dlp.exe"
if (Test-Path -LiteralPath $ytDlpExe) {
New-CmdShim -Path (Join-Path $script:BinRoot "yt-dlp.cmd") -Target "..\..\ .venv\Scripts\yt-dlp.exe".Replace(" ", "")
}
}
function Test-GoReady {
return [bool](Resolve-SystemCommand -Name "go")
}
function Test-FFmpegReady {
return [bool](Resolve-SystemCommand -Name "ffmpeg")
}
function Test-YtDlpReady {
return [bool](Resolve-SystemCommand -Name "yt-dlp")
}
function Install-LocalGo {
param(
[string]$Version = "1.24.0",
[string]$ArchiveUrl = ""
)
if (Test-Path -LiteralPath (Join-Path $script:GoBin "go.exe")) {
Write-Step "로컬 Go 이미 준비됨"
return
}
if (-not $ArchiveUrl) {
$ArchiveUrl = "https://go.dev/dl/go$Version.windows-amd64.zip"
}
Ensure-Directory -Path $script:ToolsRoot
$archivePath = Join-Path $script:LocalRoot "go-$Version.windows-amd64.zip"
Write-Step "로컬 Go 다운로드: $ArchiveUrl"
Invoke-WebRequest -Uri $ArchiveUrl -OutFile $archivePath
if (Test-Path -LiteralPath $script:GoRoot) {
Remove-Item -Recurse -Force $script:GoRoot
}
Expand-Archive -LiteralPath $archivePath -DestinationPath $script:ToolsRoot -Force
}
function Install-LocalFFmpeg {
param([string]$ArchiveUrl = "https://www.gyan.dev/ffmpeg/builds/ffmpeg-release-essentials.zip")
if (Test-Path -LiteralPath (Join-Path $script:FFmpegBin "ffmpeg.exe")) {
Write-Step "로컬 ffmpeg 이미 준비됨"
return
}
Ensure-Directory -Path $script:ToolsRoot
$archivePath = Join-Path $script:LocalRoot "ffmpeg-release-essentials.zip"
Write-Step "로컬 ffmpeg 다운로드: $ArchiveUrl"
Invoke-WebRequest -Uri $ArchiveUrl -OutFile $archivePath
$extractRoot = Join-Path $script:LocalRoot "ffmpeg-extract"
if (Test-Path -LiteralPath $extractRoot) {
Remove-Item -Recurse -Force $extractRoot
}
Expand-Archive -LiteralPath $archivePath -DestinationPath $extractRoot -Force
$binCandidate = Get-ChildItem -Path $extractRoot -Recurse -Filter "ffmpeg.exe" | Select-Object -First 1
if (-not $binCandidate) {
throw "ffmpeg.exe를 압축 파일에서 찾지 못했습니다."
}
$sourceBin = Split-Path -Parent $binCandidate.FullName
if (Test-Path -LiteralPath $script:FFmpegRoot) {
Remove-Item -Recurse -Force $script:FFmpegRoot
}
Ensure-Directory -Path $script:FFmpegBin
Copy-Item -Path (Join-Path $sourceBin "*") -Destination $script:FFmpegBin -Recurse -Force
}
function Get-AppEnvironment {
param(
[Parameter(Mandatory = $true)][string]$WorkspaceRoot,
[Parameter(Mandatory = $true)][string]$DownloadsDir,
[Parameter(Mandatory = $true)][string]$SqlitePath,
[Parameter(Mandatory = $true)][string]$AppAddr,
[string]$SearxUrl = "http://127.0.0.1:18080"
)
return @{
APP_ROOT = $WorkspaceRoot
APP_ADDR = $AppAddr
SQLITE_PATH = $SqlitePath
DOWNLOADS_DIR = $DownloadsDir
FRONTEND_DIR = (Join-Path $WorkspaceRoot "frontend")
WORKER_SCRIPT = (Join-Path $WorkspaceRoot "worker\downloader.py")
SEARXNG_BASE_URL = $SearxUrl
SEARXNG_GOOGLE_VIDEO_ENGINE = "google videos"
SEARXNG_WEB_ENGINE = "google"
}
}
function Import-GitCredential {
$credentialFile = Get-CredentialFile
if (-not (Test-Path -LiteralPath $credentialFile)) {
throw "git 자격정보 파일이 없습니다: $credentialFile"
}
return Import-PowerShellDataFile -Path $credentialFile
}
+10
View File
@@ -0,0 +1,10 @@
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
Write-Step "로컬 개발 PATH 적용 완료"
Write-Host (" Repo : " + (Get-RepoRoot))
Write-Host (" .venv : " + (Get-VenvRoot))
Write-Host (" .tools : " + (Get-ToolsRoot))
+31
View File
@@ -0,0 +1,31 @@
param(
[string]$Remote = "origin",
[string]$Branch = "",
[switch]$SetUpstream
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
$credential = Import-GitCredential
if (-not $Branch) {
$Branch = (& git branch --show-current).Trim()
}
if (-not $Branch) {
throw "현재 브랜치를 찾지 못했습니다."
}
$pair = "{0}:{1}" -f $credential.Username, $credential.Password
$token = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($pair))
$gitArgs = @("-c", "http.extraheader=AUTHORIZATION: Basic $token", "push")
if ($SetUpstream) {
$gitArgs += @("--set-upstream", $Remote, $Branch)
} else {
$gitArgs += @($Remote, $Branch)
}
Write-Step "git push $Remote $Branch"
& git @gitArgs
+28
View File
@@ -0,0 +1,28 @@
param(
[string]$AppAddr = "127.0.0.1:8080",
[string]$SqlitePath = "",
[string]$DownloadsDir = "",
[string]$SearxUrl = "http://127.0.0.1:18080"
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
$repoRoot = Get-RepoRoot
if (-not $SqlitePath) {
$SqlitePath = Join-Path $repoRoot "db\media.dev.db"
}
if (-not $DownloadsDir) {
$DownloadsDir = Join-Path $repoRoot "downloads"
}
$envMap = Get-AppEnvironment -WorkspaceRoot $repoRoot -DownloadsDir $DownloadsDir -SqlitePath $SqlitePath -AppAddr $AppAddr -SearxUrl $SearxUrl
foreach ($entry in $envMap.GetEnumerator()) {
Set-Item -Path ("Env:" + $entry.Key) -Value $entry.Value
}
Write-Step "백엔드 실행"
Invoke-CheckedCommand -FilePath "go" -Arguments @("run", "./backend")
+158
View File
@@ -0,0 +1,158 @@
param(
[int]$MockPort = 18080,
[int]$AppPort = 18081,
[switch]$SkipGoFmt
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
$repoRoot = Get-RepoRoot
$tmpRoot = Join-Path (Get-LocalRoot) ("selftest-" + [guid]::NewGuid().ToString("N"))
Ensure-Directory -Path $tmpRoot
$mockProcess = $null
$appProcess = $null
$appStdoutTask = $null
$appStderrTask = $null
function Stop-TrackedProcess {
param($Process)
if ($null -ne $Process -and -not $Process.HasExited) {
Stop-Process -Id $Process.Id -Force
$Process.WaitForExit()
}
}
try {
if (-not $SkipGoFmt) {
Write-Step "gofmt"
Invoke-CheckedCommand -FilePath "gofmt" -Arguments @(
"-w",
(Join-Path $repoRoot "backend\main.go"),
(Join-Path $repoRoot "backend\handlers\api.go"),
(Join-Path $repoRoot "backend\models\db.go"),
(Join-Path $repoRoot "backend\services\cse.go"),
(Join-Path $repoRoot "backend\services\cse_test.go"),
(Join-Path $repoRoot "backend\services\search_collectors.go"),
(Join-Path $repoRoot "backend\services\ranker.go"),
(Join-Path $repoRoot "backend\services\gemini.go"),
(Join-Path $repoRoot "backend\services\gemini_test.go")
)
}
Write-Step "python syntax"
Invoke-CheckedCommand -FilePath (Resolve-VenvPythonExe) -Arguments @(
"-m",
"py_compile",
(Join-Path $repoRoot "worker\downloader.py"),
(Join-Path $repoRoot "scripts\mock_searxng.py")
)
Write-Step "go test"
Invoke-CheckedCommand -FilePath "go" -Arguments @("test", "./...")
Write-Step "frontend syntax"
Invoke-CheckedCommand -FilePath "node" -Arguments @("--check", (Join-Path $repoRoot "frontend\app.js"))
Write-Step "go build"
$binaryPath = Join-Path $tmpRoot "ai-media-hub.exe"
Invoke-CheckedCommand -FilePath "go" -Arguments @("build", "-o", $binaryPath, "./backend")
Write-Step "start mock searxng"
$mockLog = Join-Path $tmpRoot "mock-searxng.stdout.log"
$mockErrLog = Join-Path $tmpRoot "mock-searxng.stderr.log"
$mockProcess = Start-Process -FilePath (Resolve-VenvPythonExe) `
-ArgumentList @((Join-Path $repoRoot "scripts\mock_searxng.py"), "--port", $MockPort) `
-RedirectStandardOutput $mockLog `
-RedirectStandardError $mockErrLog `
-PassThru `
-WindowStyle Hidden
Write-Step "start app"
$downloadsDir = Join-Path $tmpRoot "downloads"
Ensure-Directory -Path $downloadsDir
$appLog = Join-Path $tmpRoot "app.log"
$envMap = Get-AppEnvironment `
-WorkspaceRoot $repoRoot `
-DownloadsDir $downloadsDir `
-SqlitePath (Join-Path $tmpRoot "media.db") `
-AppAddr ("127.0.0.1:{0}" -f $AppPort) `
-SearxUrl ("http://127.0.0.1:{0}" -f $MockPort)
$appStartInfo = New-Object System.Diagnostics.ProcessStartInfo
$appStartInfo.FileName = $binaryPath
$appStartInfo.WorkingDirectory = $repoRoot
$appStartInfo.UseShellExecute = $false
$appStartInfo.RedirectStandardOutput = $true
$appStartInfo.RedirectStandardError = $true
foreach ($entry in $envMap.GetEnumerator()) {
$appStartInfo.Environment[$entry.Key] = [string]$entry.Value
}
$appProcess = New-Object System.Diagnostics.Process
$appProcess.StartInfo = $appStartInfo
$null = $appProcess.Start()
$appStdoutTask = $appProcess.StandardOutput.ReadToEndAsync()
$appStderrTask = $appProcess.StandardError.ReadToEndAsync()
$healthUrl = "http://127.0.0.1:$AppPort/healthz"
$healthy = $false
for ($i = 0; $i -lt 30; $i++) {
try {
$health = Invoke-RestMethod -Uri $healthUrl -TimeoutSec 2
if ($health.status -eq "ok") {
$healthy = $true
break
}
} catch {
Start-Sleep -Seconds 1
}
}
if (-not $healthy) {
throw "/healthz 확인 실패"
}
Write-Step "verify search"
$searchPayload = @{
query = "city rain"
platforms = @("envato", "artgrid", "google video")
} | ConvertTo-Json
$search = Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:$AppPort/api/search" -ContentType "application/json" -Body $searchPayload
$searchResults = @($search.results)
if ($null -eq $search.results -or $searchResults.Count -lt 2) {
throw "검색 결과가 너무 적습니다."
}
if (@($searchResults | Where-Object { -not $_.link }).Count -gt 0) {
throw "검색 결과에 link가 비어 있습니다."
}
Write-Step "verify upload"
$sampleFile = Join-Path $tmpRoot "sample.txt"
Set-Content -LiteralPath $sampleFile -Value "selftest upload" -Encoding UTF8
$form = @{
file = Get-Item -LiteralPath $sampleFile
}
$upload = Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:$AppPort/api/upload" -Form $form
if (-not $upload.filename) {
throw "업로드 응답에 filename이 없습니다."
}
$uploadedPath = Join-Path $downloadsDir $upload.filename
if (-not (Test-Path -LiteralPath $uploadedPath)) {
throw "업로드된 파일이 downloads에 존재하지 않습니다."
}
Write-Step "selftest ok"
} finally {
Stop-TrackedProcess -Process $appProcess
Stop-TrackedProcess -Process $mockProcess
if ($appStdoutTask) {
$appStdoutTask.Wait()
$appStdoutTask.Result | Set-Content -LiteralPath (Join-Path $tmpRoot "app.log") -Encoding UTF8
}
if ($appStderrTask) {
$appStderrTask.Wait()
$appStderrTask.Result | Add-Content -LiteralPath (Join-Path $tmpRoot "app.log") -Encoding UTF8
}
}
+43
View File
@@ -0,0 +1,43 @@
param(
[switch]$SkipPythonDeps,
[switch]$SkipGoDownload,
[switch]$SkipFFmpegDownload,
[switch]$UpgradePip
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
Ensure-Directory -Path (Get-ToolsRoot)
Ensure-Directory -Path (Get-LocalRoot)
$pythonExe = Ensure-Venv
Write-Step "Python 사용 경로: $pythonExe"
if (-not $SkipPythonDeps) {
Install-PythonRequirements -UpgradePip:$UpgradePip
}
Ensure-CommandShims
if (-not $SkipGoDownload -and -not (Test-GoReady)) {
Install-LocalGo
}
if (-not $SkipFFmpegDownload -and -not (Test-FFmpegReady)) {
Install-LocalFFmpeg
}
Use-LocalTooling
Write-Step "준비 상태"
Write-Host (" python3: " + ((Resolve-SystemCommand -Name "python3") ?? "미설치"))
Write-Host (" python : " + ((Resolve-SystemCommand -Name "python") ?? "미설치"))
Write-Host (" yt-dlp : " + ((Resolve-SystemCommand -Name "yt-dlp") ?? "미설치"))
Write-Host (" go : " + ((Resolve-SystemCommand -Name "go") ?? "미설치"))
Write-Host (" ffmpeg : " + ((Resolve-SystemCommand -Name "ffmpeg") ?? "미설치"))
Write-Step "현재 셸에 PATH를 적용하려면 다음처럼 dot-source 하세요."
Write-Host " . .\scripts\enter-dev-shell.ps1"