Compare commits

...

9 Commits

Author SHA1 Message Date
GHStaK f5d76fc3ec Replace gemini batch JSON protocol
build-push / docker (push) Successful in 4m3s
2026-03-17 17:41:33 +09:00
GHStaK b6a217cab9 Harden single-candidate gemini recovery
build-push / docker (push) Successful in 4m14s
2026-03-17 17:23:05 +09:00
GHStaK 3be797131a Reduce gemini partial batch noise
build-push / docker (push) Successful in 4m9s
2026-03-17 17:00:51 +09:00
GHStaK 513199f426 Harden gemini vision JSON recovery
build-push / docker (push) Successful in 4m13s
2026-03-17 16:33:09 +09:00
GHStaK 91ee37593c Fix gemini candidate starvation
build-push / docker (push) Successful in 4m7s
2026-03-17 16:06:59 +09:00
GHStaK 139e8f8781 Add Windows PowerShell dev workflow
build-push / docker (push) Successful in 4m7s
2026-03-17 15:47:01 +09:00
AI Assistant 75f1bb360c Reduce noisy Gemini partial failure counts
build-push / docker (push) Successful in 4m8s
2026-03-17 15:09:23 +09:00
AI Assistant 58d54a0338 Fix 504 after restoring broader search baseline
build-push / docker (push) Successful in 4m21s
2026-03-17 14:07:33 +09:00
AI Assistant c177bae59e Reapply "Strengthen search breadth and modal fitting"
This reverts commit 3f824c4bdf.
2026-03-17 14:04:07 +09:00
18 changed files with 1354 additions and 110 deletions
+4
View File
@@ -6,3 +6,7 @@ worker/__pycache__/
*.pyc
node_modules/
dist/
.venv/
.tools/
.local/
*.log
+200 -13
View File
@@ -8,6 +8,11 @@
- how it was verified
- what is still risky or incomplete
- If a push fails or a change remains local-only, that must be written here explicitly.
- Active planning and handoff notes should be written in Korean.
- After each meaningful completed task:
- update this file in context
- push the latest git state when operationally possible
- Local git credentials used for push automation must stay in an ignored local-only file and must never be committed.
## Current State At A Glance
- Project: `ai-media-hub`
@@ -33,7 +38,9 @@
- Result modal sizing is now being constrained to the viewport, and modal-only source-summary translation is now part of the active implementation path.
- Card summaries now also translate lazily to Korean, and Gemini negative-assessment handling now drives stronger follow-up search behavior than before.
- Search preview delivery is now moving away from persistent on-disk preview caching toward live proxy / live transcode behavior, with Google Video preview reuse added to result cards and modal playback.
- The latest search-breadth / modal-fitting experiment from `5ca7aef` has been rolled back after live regression was confirmed.
- Search breadth and Gemini review budget were widened again because the latest user feedback still reported a thin visible result count even after smarter filtering.
- The codebase is now back on the broader-search / modal-fitting direction associated with `5ca7aef`, with an added Gemini deadline guard to reduce reverse-proxy `504` risk.
- Windows 11 local development now has a repo-local PowerShell bootstrap / run / self-test / push workflow built around `.venv`, `.tools`, and `.local` so the machine does not need global Go / ffmpeg / yt-dlp changes for this repo.
## Current Architecture
- `backend/main.go`
@@ -216,6 +223,7 @@
- [x] Local self-test workflow
- [x] Source-specific search collectors
- [x] Shared ranker service layer
- [x] Windows 11 PowerShell local bootstrap / self-test / push workflow
## Important Current Constraints / Known Problems
- Search backend quality is still the most fragile subsystem.
@@ -232,7 +240,9 @@
- Source Summary translation now depends on Google Translate HTTP availability; frontend silently falls back to original summary text if translation fails.
- The result modal should now stay within viewport height, but this still needs real browser confirmation on multiple short-height displays because CSS-only constraints were the source of the latest user-visible regression.
- Artgrid preview playback now has a server-side ffmpeg transcode path for `.m3u8` style preview URLs, but this trades storage savings for runtime CPU cost.
- The reverted `5ca7aef` experiment showed that simply widening collector caps and Gemini candidate count can backfire when the added candidates are weak: final visible count fell sharply even though backend raw candidate count increased.
- The provided Artgrid HTML sample still does not expose a direct preview `m3u8` or `mp4` URL by itself, and `yt-dlp` probe on the sample clip URL returned `Unsupported URL`, so fully reliable Artgrid playback still depends on live-page/runtime preview discovery succeeding elsewhere in the pipeline.
- Docker CLI is not installed in this environment, so container-image build verification still cannot be performed locally from this machine.
- Gemini batch warnings can still mix true model/output failures with harmless candidate-skip cases unless the recovery path filters those error classes before surfacing them.
- The local self-test script is better than before, but it is still a smoke test, not full integration coverage.
## Current Risks Around Search Quality
@@ -423,6 +433,24 @@
- `node` is still not installed on this machine.
- This is acceptable for the current repo because there is still no Node-based frontend build or lint workflow in-tree.
- If future frontend work adds a Node toolchain, document the exact version and setup steps here before pushing.
- Windows 11 repo-local workflow:
- `scripts/setup-dev.ps1`
- creates `.venv`
- installs `worker/requirements.txt` into the repo-local venv
- downloads repo-local Go into `.tools/go`
- downloads repo-local ffmpeg into `.tools/ffmpeg`
- creates local command shims under `.tools/bin` such as `python3.cmd`
- `scripts/enter-dev-shell.ps1`
- prepends `.venv`, `.tools/bin`, local Go, and local ffmpeg to `PATH`
- pins `GOPATH`, `GOMODCACHE`, and `GOCACHE` into `.local` so Go does not need user-profile write access
- `scripts/run-dev.ps1`
- launches the backend with Windows-friendly local environment wiring
- `scripts/selftest.ps1`
- PowerShell-native replacement for the Linux shell smoke test
- verifies `gofmt`, Python syntax, `go test ./...`, frontend syntax via `node --check`, backend build, mock SearXNG boot, `/healthz`, `/api/search`, and `/api/upload`
- `scripts/push.ps1`
- reads ignored local credentials from `.local/git-credentials.psd1`
- pushes without storing credentials in tracked files
## Local Self-Test Workflow
- Primary command:
@@ -554,11 +582,14 @@
## Highest-Value Next Steps
- [ ] Reduce `/api/search` latency further without collapsing result count
- [ ] Rebuild the reverted search-expansion work from the previous stable baseline, but only after measuring where candidate quality collapses between ranked pool and final merge
- [ ] Validate the reopened `5ca7aef` search-breadth direction against real proxy timeouts and visible result count before widening it any further
- [ ] Continue hardening Gemini batch parsing so truncated JSON and ignorable low-visual candidate failures do not inflate user-facing warning counts
- [ ] Build a repeatable repo-local bootstrap script or documented setup command set for non-root machines so fresh PC setup does not depend on shell history
- [ ] Improve Envato / Artgrid preview acquisition reliability so Gemini Vision sees real frames more often
- [ ] Browser-verify the new result modal at multiple viewport heights and confirm translated Source Summary readability on real long descriptions
- [ ] Evaluate whether the new Gemini supplemental-query generation is reducing irrelevant results on a small fixed benchmark query set
- [ ] Measure runtime cost of live Artgrid preview transcoding and decide whether bounded in-memory throttling or concurrency caps are needed
- [ ] If Artgrid playback is still required for previewless clips, investigate a browser-rendered fetch path or clip-page asset bundle analysis because static HTML + `yt-dlp` probe were both insufficient on the supplied sample
- [ ] Revisit Google Video UX:
- current YouTube embed was abandoned due error `153`
- current in-app panel is more reliable but less rich than a true embedded watch page
@@ -626,22 +657,178 @@
## Recent Change Log
- Date: `2026-03-17`
- What changed:
- Reverted commit `5ca7aef` (`Strengthen search breadth and modal fitting`) to restore the previous stable search/modal baseline.
- Revalidated the rollback state locally.
- Switched the primary multi-candidate Gemini Vision response format away from JSON and toward a compact line-based text protocol:
- `index|verdict|assessment|recommended|reason_ko|search_hint`
- Kept the older JSON parser only as a fallback path instead of the primary success path.
- Reduced Gemini Vision output-token budgets again to better match the new shorter line-based format.
- Added unit coverage for the new pipe-delimited Gemini batch parser.
- Why it changed:
- The user reported that search results became too sparse again and that `search returned partial results to avoid gateway timeout` reappeared.
- The provided log `ai-media-hub-2026-03-17T04-19-08-889Z.log` showed:
- backend candidate pool still reaching `30`
- Gemini candidate cap widened to `24`
- `visualRejectCount: 12`
- final visible result count collapsing to `5`
- That indicates the widened experiment increased weak/low-visual candidates and pushed the request back toward deadline pressure without improving visible result count.
- The user-provided log `ai-media-hub-2026-03-17T08-38-47-661Z.log` still showed all Gemini batches failing with JSON output truncated almost immediately:
- `"{\"recommend"`
- `"{\"recommendations\":[{\""`
- `"{\"recommendations\":[{\"index"`
- At that point the right fix was no longer “more JSON hardening”, but removing JSON as the primary batch transport format so completed lines can be recovered even when the tail of the response is cut off.
- How it was verified:
- `pwsh -NoProfile -File scripts/selftest.ps1`
- added Go tests for line-based Gemini batch parsing
- What is still risky or incomplete:
- If Gemini returns text that does not follow either the pipe-delimited format or the JSON fallback shape, parsing can still fail.
- The model prompt is now stricter and shorter, which improves reliability, but it can make reasons more terse than before.
- Date: `2026-03-17`
- What changed:
- Added a dedicated single-candidate Gemini recovery path that no longer asks for JSON and instead parses a tiny plain-text key/value response.
- Kept multi-candidate Gemini Vision on compact JSON, but changed sequential recovery to use the shorter plain-text format automatically through the existing `Recommend(..., []SearchResult{item})` path.
- Added unit coverage for the single-candidate plain-text parser.
- Why it changed:
- The user-provided log `ai-media-hub-2026-03-17T08-20-31-074Z.log` showed even more severe truncation:
- `"{\"recommendations\":[{\"index\":"`
- `"{\"recommendations"`
- `"{\"recommend"`
- The same log showed `sequentialRetried: 0`, which means the old single-candidate recovery path was still too verbose and was not successfully rescuing failed batches.
- How it was verified:
- `pwsh -NoProfile -File scripts/selftest.ps1`
- added Go tests for single-candidate Gemini plain-text parsing
- What is still risky or incomplete:
- If Gemini returns malformed plain text that omits the required `verdict:` line, even the single-candidate recovery path can still fail.
- This improves recovery robustness, but total Gemini latency can still rise when many batch failures fall back to candidate-by-candidate evaluation.
- Date: `2026-03-17`
- What changed:
- Added adaptive Gemini Vision output-token sizing so smaller candidate batches, especially single-candidate sequential recovery calls, now request much shorter responses.
- Added a dedicated shorter single-candidate Gemini Vision instruction path for sequential recovery after batch failure.
- Stopped counting a batch as a strong user-facing partial failure when sequential recovery still salvages recommendations from that batch.
- Added unit coverage for the adaptive Gemini Vision token budget helper.
- Why it changed:
- The user-provided log `ai-media-hub-2026-03-17T07-55-17-127Z.log` still showed `gemini vision partially failed on 4 of 6 batches`.
- The same log also showed `sequentialRetried: 0`, which means the fallback single-candidate reevaluation path was still not recovering those truncated JSON batches well enough.
- How it was verified:
- `pwsh -NoProfile -File scripts/selftest.ps1`
- added Go tests for adaptive Gemini token sizing
- What is still risky or incomplete:
- This reduces partial-failure pressure further, but extremely short or malformed Gemini outputs can still fail before one complete recommendation object is emitted.
- Smaller recovery responses improve reliability, but repeated sequential recovery can still add latency on difficult searches.
- Date: `2026-03-17`
- What changed:
- Reduced Gemini Vision batch size from `6` to `4` so each model response carries fewer recommendation objects and is less likely to be truncated mid-JSON.
- Tightened the Gemini Vision prompt to ask for shorter Korean reasons and compact JSON-only output.
- Lowered Gemini Vision `maxOutputTokens` and added partial JSON recovery so already-complete recommendation objects can still be salvaged when the model output is cut off before the final closing braces.
- Added unit coverage for truncated Gemini Vision JSON recovery.
- Why it changed:
- The user-provided log `ai-media-hub-2026-03-17T07-29-44-949Z.log` showed that visuals were prepared successfully, but every batch failed with `gemini vision JSON extraction failed: no complete JSON object found ...`.
- The failure pattern indicates output truncation rather than missing thumbnails or preview frames.
- How it was verified:
- `pwsh -NoProfile -File scripts/selftest.ps1`
- added Go tests for partial Gemini JSON recovery behavior
- What is still risky or incomplete:
- If Gemini returns severely malformed output before even one full recommendation object closes, the parser still cannot recover useful results from that batch.
- Smaller batch size improves reliability but can increase total Gemini round trips and latency on some searches.
- Date: `2026-03-17`
- What changed:
- Fixed a search-budget regression where source collection could consume the full `SearchService` deadline and leave no time for Envato / Artgrid enrichment, causing Gemini to see only missing or low-value visuals.
- Split the search-service deadline into:
- collector deadline
- enrichment deadline with an explicit reserved window
- Added unit coverage for the new deadline split behavior.
- Stopped frontend preview-probe fallback from calling `/api/download/preview` for Artgrid items that do not already have a provider preview URL, so unsupported `yt-dlp` Artgrid probe errors no longer fire just from opening or hovering those results.
- Why it changed:
- The user-provided log `ai-media-hub-2026-03-17T07-01-21-282Z.log` showed:
- `search_service:deadline_reached`
- immediate `search_service:enrich_start` -> `search_service:enrich_complete`
- `withPreview: 0`
- `withLowValueThumbnail: 12`
- repeated `candidate has no thumbnail or preview video`
- final warning `gemini vision returned no candidate evaluations`
- The same log also showed Artgrid preview probe failures from `yt-dlp` returning `Unsupported URL`, which were not helping user-facing preview behavior.
- How it was verified:
- `pwsh -NoProfile -File scripts/selftest.ps1`
- added Go tests for the search/enrichment deadline split helper
- What is still risky or incomplete:
- This preserves time for enrichment, but it does not guarantee that every live Envato / Artgrid page yields a usable preview URL.
- Artgrid still depends on backend-enriched provider preview URLs for true video preview; if no provider preview is discovered, the UI will still fall back to thumbnail-only rendering.
- Date: `2026-03-17`
- What changed:
- Added repo-local Windows 11 PowerShell workflows:
- `scripts/dev-tools.ps1`
- `scripts/setup-dev.ps1`
- `scripts/enter-dev-shell.ps1`
- `scripts/run-dev.ps1`
- `scripts/selftest.ps1`
- `scripts/push.ps1`
- Added ignored local directories for `.venv`, `.tools`, and `.local`.
- Created a local-only git credential file for automated push flow and kept it excluded from git.
- Pinned PowerShell tooling to repo-local Go caches under `.local` so `go test` no longer depends on writable user-profile Go paths.
- Why it changed:
- The active development machine is now Windows 11, and the user requested that setup, verification, and push flows work through PowerShell while minimizing machine-wide side effects.
- The previous local workflow was Linux-shell-oriented and did not directly cover Windows PowerShell usage.
- Initial Windows self-test attempts failed because Go wanted to write into the default user Go cache path, which was not reliably writable in this environment.
- How it was verified:
- `pwsh -NoProfile -File scripts/setup-dev.ps1 -SkipPythonDeps -SkipGoDownload -SkipFFmpegDownload`
- elevated `pwsh -NoProfile -File scripts/setup-dev.ps1`
- `pwsh -NoProfile -Command ". .\scripts\enter-dev-shell.ps1; python3 --version; yt-dlp --version; go version; ffmpeg -version | Select-Object -First 1"`
- `pwsh -NoProfile -File scripts/selftest.ps1`
- `git check-ignore -v .local\git-credentials.psd1 .venv .tools`
- What is still risky or incomplete:
- `scripts/setup-dev.ps1` still requires network access for Python package install and local Go / ffmpeg downloads on a truly fresh machine.
- The backend runtime still invokes `python3` by name, so Windows usage depends on entering the repo-local dev shell or otherwise exposing the generated shim on `PATH`.
- Push automation now uses an ignored local credential file, but credential rotation remains a manual step.
- Date: `2026-03-17`
- What changed:
- Restored hard/ignorable Gemini batch error filtering so low-value thumbnail skips and no-visual candidate skips do not count as user-facing partial batch failures when useful recommendations were still recovered.
- Added unit coverage for the Gemini batch error filtering behavior.
- Why it changed:
- The user reported the warning `gemini vision partially failed on 3 of 4 batches`.
- The provided log `ai-media-hub-2026-03-17T06-06-04-447Z.log` showed three different batch-failure classes mixed together:
- real Gemini JSON extraction failures
- low-value thumbnail skips
- no-visual candidate skips
- Only the first class should strongly influence the user-facing partial-failure warning.
- How it was verified:
- `go test ./...`
- `bash scripts/selftest.sh`
- What is still risky or incomplete:
- This rollback restores the earlier baseline but does not solve the underlying request to improve visible result count.
- The next attempt should start from the reverted baseline and target the actual bottleneck between ranked pool and final merge, instead of only widening raw query breadth.
- This reduces noisy partial-failure warnings, but it does not eliminate genuine Gemini JSON truncation or provider-side preview problems.
- If the model starts returning different malformed JSON patterns, the parser and warning logic may still need further hardening.
- Date: `2026-03-17`
- What changed:
- Reapplied the broader-search / modal-fitting codepath from `5ca7aef` as requested by the user.
- Added a targeted 504 mitigation by making Gemini batch recovery stop at the request deadline instead of continuing sequential single-candidate retries indefinitely.
- Kept an explicit Gemini vision JSON output token cap to reduce the chance of truncated model responses during larger structured batches.
- Why it changed:
- The user explicitly asked to return to the `5ca7aef` direction and then fix the live `504 Gateway Time-out`.
- The provided log `ai-media-hub-2026-03-17T04-59-24-566Z.log` showed backend search collection finishing in about `22s`, then Gemini batch failure plus sequential recovery continuing until the reverse proxy returned `504`.
- The direct cause was deadline-unaware Gemini recovery work, not initial search collection itself.
- How it was verified:
- `go test ./...`
- `bash scripts/selftest.sh`
- attempted `docker build -t ai-media-hub:test .` but `docker` is not installed in this environment
- What is still risky or incomplete:
- This reduces one concrete `504` path, but it does not guarantee that all reverse-proxy timeout cases are eliminated under worse upstream conditions.
- Container-image build verification still needs to happen on a machine that actually has Docker available.
- Date: `2026-03-17`
- What changed:
- Raised search breadth again by widening per-source collector caps, increasing the number of base queries considered, increasing per-collector query budgets, and expanding the Gemini candidate review budget.
- Added another browser-resolution-aware modal fitting pass: the result popup now recalculates shell width, shell height, media max height, and compact-mode behavior from `window.innerWidth` / `window.innerHeight` when opened or when the viewport changes.
- Removed the fixed minimum heights on the lower modal panels so they can shrink with the viewport instead of forcing the popup past the visible browser area.
- Checked the supplied Artgrid clip HTML and a direct `yt-dlp` probe against `https://artgrid.io/clip/355470/...`; the HTML still exposes metadata and thumbnail signals only, and the direct probe returned `Unsupported URL`.
- Why it changed:
- The user reported that the popup still overflowed the browser viewport and that the visible result set still felt too thin despite the earlier search-expansion work.
- The attached Artgrid HTML sample was intended to verify whether a stable preview URL could be extracted directly from static clip HTML.
- How it was verified:
- `go test ./...`
- `bash scripts/selftest.sh`
- `python3 -m py_compile worker/downloader.py scripts/mock_searxng.py`
- `python3 worker/downloader.py --mode probe --url 'https://artgrid.io/clip/355470/couple-woman-man-holding-hand'`
- What is still risky or incomplete:
- The new modal layout now adapts to actual browser viewport dimensions in code, but a live browser confirmation on the exact user display is still needed.
- Widening search breadth and Gemini review budget should improve visible count, but it can also increase latency and token cost under poor upstream conditions.
- The supplied Artgrid HTML sample alone was not enough to derive a directly playable preview URL, so previewless Artgrid clips may still fall back to thumbnail-only behavior.
- Date: `2026-03-17`
- What changed:
+27 -24
View File
@@ -554,7 +554,7 @@ func (a *App) searchMedia(c *gin.Context) {
return
}
targetCount := 16
targetCount := 18
merged := services.MergeRecommendations(recommended, scored, targetCount)
if geminiErr != nil {
merged = services.BackfillRecommendations(
@@ -564,31 +564,34 @@ func (a *App) searchMedia(c *gin.Context) {
"Gemini 배치 일부가 실패해 미리보기 가능한 상위 후보를 보강했습니다.",
)
}
if len(merged) < targetCount && time.Now().Before(deadline.Add(-5*time.Second)) {
for pass := 0; pass < 2 && len(merged) < targetCount && time.Now().Before(deadline.Add(-4*time.Second)); pass++ {
coverageQueries := buildCoverageQueries(req.Query, queryVariants, recommended, merged)
if len(coverageQueries) > 0 {
a.debug("search coverage query variants", gin.H{"variants": coverageQueries, "variantCount": len(coverageQueries), "existingCount": len(merged)})
extraResults, extraMeta, extraErr := a.SearchService.SearchMediaWithDeadline(coverageQueries, enabledPlatforms, deadline.Add(-5*time.Second))
supplementalDeadlineLimited = supplementalDeadlineLimited || extraMeta.PartialDueToDeadline
if extraErr == nil && len(extraResults) > 0 {
results = mergeSearchResults(results, extraResults)
scored = services.RankSearchResults(strings.Join(coverageQueries[:min(len(coverageQueries), 3)], " "), results)
reviewedLinks := services.ReviewedRecommendationLinks(recommended)
supplementalCandidates := services.SelectUnevaluatedCandidates(scored, reviewedLinks, services.RemainingGeminiCapacity(recommended))
if len(supplementalCandidates) > 0 {
extraRecommended, extraStats, extraGeminiErr := services.EvaluateAllCandidatesWithGeminiWithDeadline(
a.GeminiService,
req.Query,
supplementalCandidates,
deadline.Add(-2*time.Second),
)
recommended = services.MergeUniqueRecommendations(recommended, extraRecommended)
geminiStats = services.MergeGeminiBatchStats(geminiStats, extraStats)
geminiErr = combineSearchWarnings(geminiErr, extraGeminiErr)
}
merged = services.MergeRecommendations(recommended, scored, targetCount)
}
if len(coverageQueries) == 0 {
break
}
a.debug("search coverage query variants", gin.H{"variants": coverageQueries, "variantCount": len(coverageQueries), "existingCount": len(merged), "pass": pass + 1})
extraResults, extraMeta, extraErr := a.SearchService.SearchMediaWithDeadline(coverageQueries, enabledPlatforms, deadline.Add(-4*time.Second))
supplementalDeadlineLimited = supplementalDeadlineLimited || extraMeta.PartialDueToDeadline
if extraErr != nil || len(extraResults) == 0 {
break
}
results = mergeSearchResults(results, extraResults)
scored = services.RankSearchResults(strings.Join(coverageQueries[:min(len(coverageQueries), 3)], " "), results)
reviewedLinks := services.ReviewedRecommendationLinks(recommended)
supplementalCandidates := services.SelectUnevaluatedCandidates(scored, reviewedLinks, services.RemainingGeminiCapacity(recommended))
if len(supplementalCandidates) == 0 {
break
}
extraRecommended, extraStats, extraGeminiErr := services.EvaluateAllCandidatesWithGeminiWithDeadline(
a.GeminiService,
req.Query,
supplementalCandidates,
deadline.Add(-2*time.Second),
)
recommended = services.MergeUniqueRecommendations(recommended, extraRecommended)
geminiStats = services.MergeGeminiBatchStats(geminiStats, extraStats)
geminiErr = combineSearchWarnings(geminiErr, extraGeminiErr)
merged = services.MergeRecommendations(recommended, scored, targetCount)
}
if len(merged) < targetCount {
merged = services.BackfillRecommendations(merged, scored, targetCount, "추가 검색 후에도 충분한 결과가 부족해 시각 자산이 있는 후보를 제한적으로 보강했습니다.")
+23 -8
View File
@@ -53,6 +53,8 @@ type SearchExecutionMeta struct {
PartialDueToDeadline bool `json:"partialDueToDeadline"`
}
const searchEnrichmentReserve = 4 * time.Second
func NewSearchService(baseURL, googleVideoEngine, webEngine string) *SearchService {
if googleVideoEngine == "" {
googleVideoEngine = "google videos"
@@ -84,9 +86,11 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
if s.BaseURL == "" {
return nil, meta, fmt.Errorf("searxng base url is not configured")
}
collectionDeadline, enrichmentDeadline := splitSearchDeadlines(deadline)
s.debug("search_service:start", map[string]any{
"queries": queries,
"enabledPlatforms": enabledPlatforms,
"deadlineSet": !deadline.IsZero(),
})
seen := map[string]bool{}
@@ -94,12 +98,12 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
results := make([]SearchResult, 0, 90)
var lastErr error
baseQueries := limitQueries(queries, 8)
baseQueries := limitQueries(queries, 10)
shuffleStrings(baseQueries)
primaryQueries := baseQueries[:minInt(len(baseQueries), 3)]
primaryQueries := baseQueries[:minInt(len(baseQueries), 4)]
runSearchPass := func(bases []string, onlyMissing bool) {
for _, base := range bases {
if !deadline.IsZero() && time.Now().After(deadline) {
if !collectionDeadline.IsZero() && time.Now().After(collectionDeadline) {
meta.PartialDueToDeadline = true
s.debug("search_service:deadline_reached", map[string]any{"stage": "runSearchPass", "base": base})
return
@@ -109,7 +113,7 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
continue
}
for _, collector := range s.collectors {
if !deadline.IsZero() && time.Now().After(deadline) {
if !collectionDeadline.IsZero() && time.Now().After(collectionDeadline) {
meta.PartialDueToDeadline = true
s.debug("search_service:deadline_reached", map[string]any{"stage": "collectorLoop", "collector": collector.Name()})
return
@@ -133,7 +137,7 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
"searchQueries": searchQueries,
})
for _, searchQuery := range searchQueries {
if !deadline.IsZero() && time.Now().After(deadline) {
if !collectionDeadline.IsZero() && time.Now().After(collectionDeadline) {
meta.PartialDueToDeadline = true
s.debug("search_service:deadline_reached", map[string]any{"stage": "queryLoop", "collector": collector.Name(), "query": searchQuery})
return
@@ -192,11 +196,22 @@ func (s *SearchService) SearchMediaWithDeadline(queries []string, enabledPlatfor
"hadError": lastErr != nil,
"partialDueToDeadline": meta.PartialDueToDeadline,
})
enriched, enrichMeta := s.EnrichResultsWithDeadline(results, deadline)
enriched, enrichMeta := s.EnrichResultsWithDeadline(results, enrichmentDeadline)
meta.PartialDueToDeadline = meta.PartialDueToDeadline || enrichMeta.PartialDueToDeadline
return enriched, meta, nil
}
func splitSearchDeadlines(deadline time.Time) (time.Time, time.Time) {
if deadline.IsZero() {
return time.Time{}, time.Time{}
}
remaining := time.Until(deadline)
if remaining <= searchEnrichmentReserve {
return deadline, deadline
}
return deadline.Add(-searchEnrichmentReserve), deadline
}
func (s *SearchService) EnrichResults(results []SearchResult) []SearchResult {
enriched, _ := s.EnrichResultsWithDeadline(results, time.Time{})
return enriched
@@ -1432,9 +1447,9 @@ func limitCollectorQueries(collector string, queries []string, onlyMissing bool)
limit := 2
switch collector {
case "Envato", "Artgrid":
limit = 4
limit = 5
case "Google Video":
limit = 3
limit = 4
}
if onlyMissing {
limit--
+28 -4
View File
@@ -159,13 +159,13 @@ func TestLimitCollectorQueriesUsesSmallerBudgetForMissingPass(t *testing.T) {
queries := []string{"a", "b", "c", "d"}
got := limitCollectorQueries("Artgrid", queries, true)
if len(got) != 3 {
t.Fatalf("expected 3 queries for missing-pass Artgrid collector, got %d", len(got))
if len(got) != 4 {
t.Fatalf("expected 4 queries for missing-pass Artgrid collector, got %d", len(got))
}
got = limitCollectorQueries("Google Video", queries, false)
if len(got) != 3 {
t.Fatalf("expected 3 queries for Google Video collector, got %d", len(got))
if len(got) != 4 {
t.Fatalf("expected 4 queries for Google Video collector, got %d", len(got))
}
}
@@ -182,6 +182,30 @@ func TestSearchServiceFetchCacheRoundTrip(t *testing.T) {
}
}
func TestSplitSearchDeadlinesReservesEnrichmentWindow(t *testing.T) {
deadline := time.Now().Add(20 * time.Second)
collectionDeadline, enrichmentDeadline := splitSearchDeadlines(deadline)
if enrichmentDeadline.IsZero() {
t.Fatal("expected enrichment deadline to be preserved")
}
if !collectionDeadline.Before(enrichmentDeadline) {
t.Fatalf("expected collection deadline before enrichment deadline, got %v >= %v", collectionDeadline, enrichmentDeadline)
}
if gap := enrichmentDeadline.Sub(collectionDeadline); gap < searchEnrichmentReserve-500*time.Millisecond {
t.Fatalf("expected reserve close to %v, got %v", searchEnrichmentReserve, gap)
}
}
func TestSplitSearchDeadlinesDoesNotReserveWhenDeadlineIsTooClose(t *testing.T) {
deadline := time.Now().Add(2 * time.Second)
collectionDeadline, enrichmentDeadline := splitSearchDeadlines(deadline)
if !collectionDeadline.Equal(enrichmentDeadline) {
t.Fatalf("expected identical deadlines when budget is too tight, got %v and %v", collectionDeadline, enrichmentDeadline)
}
}
func TestSearchServiceSkipsArtgridAPIAfter403(t *testing.T) {
var apiRequests atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+348 -34
View File
@@ -13,6 +13,7 @@ import (
"os/exec"
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"time"
@@ -245,6 +246,9 @@ func (g *GeminiService) Recommend(query string, candidates []SearchResult) ([]AI
if len(candidates) == 0 {
return []AIRecommendation{}, nil
}
if len(candidates) == 1 {
return g.recommendSingleCandidate(query, candidates[0])
}
g.debug("gemini:vision_start", map[string]any{
"query": query,
"candidateCount": len(candidates),
@@ -253,22 +257,7 @@ func (g *GeminiService) Recommend(query string, candidates []SearchResult) ([]AI
type geminiPart map[string]any
parts := []geminiPart{
{
"text": `You are a professional video editor. Analyze whether each provided visual is suitable as a usable scene or shot for the user's requested keyword. Return JSON only in this shape:
{"recommendations":[{"index":0,"verdict":"Yes","reason":"short reason","recommended":true,"assessment":"positive","searchHint":"short english hint"}]}
Return one entry for every analyzed candidate. Use Korean for every reason. Keep reasons concise but specific enough to explain usefulness.
Set verdict to "Yes" or "No" for every candidate. "Yes" means the scene is usable and relevant for editing against the user's keyword. "No" means it is not suitable or not relevant enough.
Set recommended=true only when verdict is "Yes". Set recommended=false when verdict is "No".
Set assessment to one of: positive, unclear, irrelevant, inappropriate.
- positive: directly usable and relevant to the query
- unclear: visually ambiguous, weak, or not confident enough
- irrelevant: visibly unrelated to the query intent
- inappropriate: low-quality, spammy, misleading, meme-like, or otherwise unsuitable for professional editing
When assessment is not positive, provide searchHint as a short English stock-footage search phrase that could help find better candidates. Keep it under 8 words.
When assessment is positive, searchHint may be empty.
Prefer cinematic b-roll, stock footage, editorial footage, clean composition, usable establishing shots, and professional media thumbnails.
Avoid clickbait faces, exaggerated expressions, meme aesthetics, low-information thumbnails, sensational text overlays, or gossip-style imagery.
Favor scenes that look directly useful for professional editing, sequencing, establishing, cutaway, or mood-building usage.
User query: ` + query,
"text": buildGeminiVisionInstruction(query, len(candidates)),
},
}
@@ -295,9 +284,10 @@ User query: ` + query,
return nil, fmt.Errorf("no candidate thumbnails or preview frames could be fetched for gemini vision")
}
g.debug("gemini:vision_visuals_prepared", map[string]any{
"query": query,
"visualCount": visualCount,
"maxImages": maxImages,
"query": query,
"visualCount": visualCount,
"maxImages": maxImages,
"maxOutputTokens": geminiVisionMaxOutputTokens(visualCount),
})
body := map[string]any{
@@ -305,7 +295,9 @@ User query: ` + query,
{"parts": parts},
},
"generationConfig": map[string]any{
"responseMimeType": "application/json",
"responseMimeType": "text/plain",
"temperature": 0.1,
"maxOutputTokens": geminiVisionMaxOutputTokens(visualCount),
},
}
@@ -338,23 +330,17 @@ User query: ` + query,
return nil, fmt.Errorf("gemini vision returned no candidates")
}
jsonText, err := extractJSONObject(payload.Candidates[0].Content.Parts[0].Text)
rawText := payload.Candidates[0].Content.Parts[0].Text
parsed, recoveredPartial, err := parseGeminiVisionRecommendations(rawText)
if err != nil {
return nil, fmt.Errorf("gemini vision JSON extraction failed: %w", err)
}
var parsed struct {
Recommendations []struct {
Index int `json:"index"`
Verdict string `json:"verdict"`
Reason string `json:"reason"`
Recommended bool `json:"recommended"`
Assessment string `json:"assessment"`
SearchHint string `json:"searchHint"`
} `json:"recommendations"`
}
if err := json.Unmarshal([]byte(jsonText), &parsed); err != nil {
return nil, fmt.Errorf("gemini vision JSON parse failed: %w; raw=%q", err, truncateForError(payload.Candidates[0].Content.Parts[0].Text, 200))
if recoveredPartial {
g.debug("gemini:vision_partial_json_recovered", map[string]any{
"query": query,
"candidateCount": len(candidates),
"recommendationCount": len(parsed.Recommendations),
})
}
recommendations := make([]AIRecommendation, 0, len(parsed.Recommendations))
@@ -386,6 +372,334 @@ User query: ` + query,
return recommendations, nil
}
func (g *GeminiService) recommendSingleCandidate(query string, candidate SearchResult) ([]AIRecommendation, error) {
g.debug("gemini:vision_start", map[string]any{
"query": query,
"candidateCount": 1,
"mode": "single_candidate_recovery",
})
img, mimeType, err := g.fetchCandidateVisualInlineData(candidate)
if err != nil {
g.debug("gemini:vision_candidate_visual_error", map[string]any{
"index": 0,
"link": candidate.Link,
"source": candidate.Source,
"error": err.Error(),
})
return nil, err
}
g.debug("gemini:vision_visuals_prepared", map[string]any{
"query": query,
"visualCount": 1,
"maxImages": 1,
"maxOutputTokens": 120,
"mode": "single_candidate_recovery",
})
body := map[string]any{
"contents": []map[string]any{
{
"parts": []map[string]any{
{
"text": `You are a professional video editor. Analyze the single provided visual for the user's keyword.
Return plain text only with exactly these 5 lines:
verdict: Yes or No
assessment: positive or unclear or irrelevant or inappropriate
recommended: true or false
reason_ko: very short Korean reason
search_hint: short English stock-footage hint or empty
No JSON. No markdown. No extra text.
User query: ` + query,
},
{"text": fmt.Sprintf("Candidate 0: title=%s source=%s link=%s", candidate.Title, candidate.Source, candidate.Link)},
{"inlineData": map[string]string{"mimeType": mimeType, "data": img}},
},
},
},
"generationConfig": map[string]any{
"responseMimeType": "text/plain",
"temperature": 0.1,
"maxOutputTokens": 120,
},
}
rawText, err := g.generateText(body)
if err != nil {
return nil, err
}
rec, err := parseSingleCandidateVisionText(rawText)
if err != nil {
return nil, fmt.Errorf("gemini single-candidate parse failed: %w; raw=%q", err, truncateForError(rawText, 200))
}
recommended := rec.Recommended || strings.EqualFold(strings.TrimSpace(rec.Verdict), "yes")
assessment := normalizeAssessment(rec.Assessment, recommended)
result := AIRecommendation{
Title: candidate.Title,
Link: candidate.Link,
Snippet: candidate.Snippet,
ThumbnailURL: candidate.ThumbnailURL,
PreviewVideoURL: candidate.PreviewVideoURL,
Source: candidate.Source,
Reason: normalizeKoreanReason(rec.Reason),
Recommended: recommended,
Assessment: assessment,
SearchHint: normalizeSearchHint(rec.SearchHint),
}
g.debug("gemini:vision_complete", map[string]any{
"query": query,
"recommendationCount": 1,
"mode": "single_candidate_recovery",
})
return []AIRecommendation{result}, nil
}
func buildGeminiVisionInstruction(query string, _ int) string {
return `You are a professional video editor. Analyze whether each provided visual is suitable as a usable scene or shot for the user's requested keyword.
Return plain text only.
Return exactly one line per analyzed candidate in this exact format:
index|verdict|assessment|recommended|reason_ko|search_hint
Rules:
- index: integer candidate index
- verdict: Yes or No
- assessment: positive or unclear or irrelevant or inappropriate
- recommended: true or false
- reason_ko: very short Korean reason without line breaks and without |
- search_hint: short English stock-footage phrase or empty, without |
Do not include markdown fences, JSON, bullets, numbering, or any other text.
Prefer cinematic b-roll, stock footage, editorial footage, clean composition, usable establishing shots, and professional media thumbnails.
Avoid clickbait faces, exaggerated expressions, meme aesthetics, low-information thumbnails, sensational text overlays, or gossip-style imagery.
User query: ` + query
}
func geminiVisionMaxOutputTokens(candidateCount int) int {
switch {
case candidateCount <= 1:
return 120
case candidateCount == 2:
return 180
case candidateCount == 3:
return 240
case candidateCount == 4:
return 300
default:
return 360
}
}
type geminiVisionParsedPayload struct {
Recommendations []struct {
Index int `json:"index"`
Verdict string `json:"verdict"`
Reason string `json:"reason"`
Recommended bool `json:"recommended"`
Assessment string `json:"assessment"`
SearchHint string `json:"searchHint"`
} `json:"recommendations"`
}
func parseGeminiVisionRecommendations(raw string) (geminiVisionParsedPayload, bool, error) {
if parsed, ok := parseGeminiVisionLines(raw); ok {
return parsed, false, nil
}
jsonText, err := extractJSONObject(raw)
if err == nil {
var parsed geminiVisionParsedPayload
if unmarshalErr := json.Unmarshal([]byte(jsonText), &parsed); unmarshalErr != nil {
return geminiVisionParsedPayload{}, false, fmt.Errorf("json parse failed: %w; raw=%q", unmarshalErr, truncateForError(raw, 200))
}
return parsed, false, nil
}
objects := extractCompleteRecommendationObjects(raw)
if len(objects) == 0 {
return geminiVisionParsedPayload{}, false, err
}
parsed := geminiVisionParsedPayload{
Recommendations: make([]struct {
Index int `json:"index"`
Verdict string `json:"verdict"`
Reason string `json:"reason"`
Recommended bool `json:"recommended"`
Assessment string `json:"assessment"`
SearchHint string `json:"searchHint"`
}, 0, len(objects)),
}
for _, objectText := range objects {
var item struct {
Index int `json:"index"`
Verdict string `json:"verdict"`
Reason string `json:"reason"`
Recommended bool `json:"recommended"`
Assessment string `json:"assessment"`
SearchHint string `json:"searchHint"`
}
if unmarshalErr := json.Unmarshal([]byte(objectText), &item); unmarshalErr != nil {
continue
}
parsed.Recommendations = append(parsed.Recommendations, item)
}
if len(parsed.Recommendations) == 0 {
return geminiVisionParsedPayload{}, false, err
}
return parsed, true, nil
}
func parseGeminiVisionLines(raw string) (geminiVisionParsedPayload, bool) {
lines := strings.Split(strings.ReplaceAll(strings.TrimSpace(raw), "\r\n", "\n"), "\n")
parsed := geminiVisionParsedPayload{
Recommendations: make([]struct {
Index int `json:"index"`
Verdict string `json:"verdict"`
Reason string `json:"reason"`
Recommended bool `json:"recommended"`
Assessment string `json:"assessment"`
SearchHint string `json:"searchHint"`
}, 0, len(lines)),
}
for _, line := range lines {
trimmed := strings.TrimSpace(strings.Trim(line, "`"))
if trimmed == "" {
continue
}
parts := strings.SplitN(trimmed, "|", 6)
if len(parts) != 6 {
continue
}
index, err := strconv.Atoi(strings.TrimSpace(parts[0]))
if err != nil {
continue
}
parsed.Recommendations = append(parsed.Recommendations, struct {
Index int `json:"index"`
Verdict string `json:"verdict"`
Reason string `json:"reason"`
Recommended bool `json:"recommended"`
Assessment string `json:"assessment"`
SearchHint string `json:"searchHint"`
}{
Index: index,
Verdict: strings.TrimSpace(parts[1]),
Assessment: strings.TrimSpace(parts[2]),
Recommended: strings.EqualFold(strings.TrimSpace(parts[3]), "true") || strings.EqualFold(strings.TrimSpace(parts[3]), "yes"),
Reason: strings.TrimSpace(parts[4]),
SearchHint: strings.TrimSpace(parts[5]),
})
}
return parsed, len(parsed.Recommendations) > 0
}
type singleCandidateVisionResponse struct {
Verdict string
Assessment string
Recommended bool
Reason string
SearchHint string
}
func parseSingleCandidateVisionText(raw string) (singleCandidateVisionResponse, error) {
lines := strings.Split(strings.ReplaceAll(strings.TrimSpace(raw), "\r\n", "\n"), "\n")
result := singleCandidateVisionResponse{}
for _, line := range lines {
trimmed := strings.TrimSpace(line)
if trimmed == "" {
continue
}
parts := strings.SplitN(trimmed, ":", 2)
if len(parts) != 2 {
continue
}
key := strings.ToLower(strings.TrimSpace(parts[0]))
value := strings.TrimSpace(parts[1])
switch key {
case "verdict":
result.Verdict = value
case "assessment":
result.Assessment = value
case "recommended":
result.Recommended = strings.EqualFold(value, "true") || strings.EqualFold(value, "yes")
case "reason_ko":
result.Reason = value
case "search_hint":
result.SearchHint = value
}
}
if strings.TrimSpace(result.Verdict) == "" {
return singleCandidateVisionResponse{}, fmt.Errorf("missing verdict line")
}
return result, nil
}
func extractCompleteRecommendationObjects(text string) []string {
cleaned := strings.TrimSpace(text)
cleaned = strings.TrimPrefix(cleaned, "```json")
cleaned = strings.TrimPrefix(cleaned, "```")
cleaned = strings.TrimSuffix(cleaned, "```")
cleaned = strings.TrimSpace(cleaned)
recommendationsIndex := strings.Index(cleaned, `"recommendations"`)
if recommendationsIndex == -1 {
return nil
}
arrayStart := strings.Index(cleaned[recommendationsIndex:], "[")
if arrayStart == -1 {
return nil
}
arrayStart += recommendationsIndex
objects := make([]string, 0, 4)
inString := false
escaped := false
objectDepth := 0
objectStart := -1
for idx := arrayStart + 1; idx < len(cleaned); idx++ {
ch := cleaned[idx]
if escaped {
escaped = false
continue
}
if ch == '\\' && inString {
escaped = true
continue
}
if ch == '"' {
inString = !inString
continue
}
if inString {
continue
}
switch ch {
case '{':
if objectDepth == 0 {
objectStart = idx
}
objectDepth++
case '}':
if objectDepth == 0 {
continue
}
objectDepth--
if objectDepth == 0 && objectStart >= 0 {
objects = append(objects, cleaned[objectStart:idx+1])
objectStart = -1
}
case ']':
if objectDepth == 0 {
return objects
}
}
}
return objects
}
func (g *GeminiService) BuildSupplementalQueries(query string, existing []string, reviewed []AIRecommendation) ([]string, error) {
baseExisting := make([]string, 0, len(existing))
for _, item := range existing {
+82 -2
View File
@@ -123,8 +123,8 @@ func TestRemainingGeminiCapacityShrinksWithReviewedItems(t *testing.T) {
{Link: "https://a.example"},
{Link: "https://b.example"},
}
if got := RemainingGeminiCapacity(reviewed); got != 14 {
t.Fatalf("expected 14 remaining slots, got %d", got)
if got := RemainingGeminiCapacity(reviewed); got != 22 {
t.Fatalf("expected 22 remaining slots, got %d", got)
}
}
@@ -213,3 +213,83 @@ func TestMergeRecommendationsExcludesIrrelevantAndPendingFiller(t *testing.T) {
t.Fatalf("unexpected merged result: %#v", merged)
}
}
func TestFilterHardGeminiErrorsIgnoresLowValueVisualFailures(t *testing.T) {
errs := []string{
"candidate thumbnail is low value",
"no candidate thumbnails or preview frames could be fetched for gemini vision",
"gemini vision JSON extraction failed: no complete JSON object found",
}
filtered := filterHardGeminiErrors(errs)
if len(filtered) != 1 {
t.Fatalf("expected only hard errors to remain, got %#v", filtered)
}
if !strings.Contains(filtered[0], "JSON extraction failed") {
t.Fatalf("unexpected filtered errors: %#v", filtered)
}
}
func TestParseGeminiVisionRecommendationsRecoversCompleteObjectsFromTruncatedJSON(t *testing.T) {
raw := "{\n" +
" \"recommendations\": [\n" +
" {\"index\":0,\"verdict\":\"Yes\",\"reason\":\"적합\",\"recommended\":true,\"assessment\":\"positive\",\"searchHint\":\"\"},\n" +
" {\"index\":1,\"verdict\":\"No\",\"reason\":\"부적합\",\"recommended\":false,\"assessment\":\"irrelevant\",\"searchHint\":\"night city b-roll\"},\n" +
" {\"index\":2,\"verdict\":\"Yes\",\"reason\":\"잘림"
parsed, recoveredPartial, err := parseGeminiVisionRecommendations(raw)
if err != nil {
t.Fatalf("expected partial recovery, got error: %v", err)
}
if !recoveredPartial {
t.Fatal("expected partial recovery flag to be true")
}
if len(parsed.Recommendations) != 2 {
t.Fatalf("expected 2 recovered recommendation objects, got %#v", parsed.Recommendations)
}
if parsed.Recommendations[0].Index != 0 || parsed.Recommendations[1].Index != 1 {
t.Fatalf("unexpected recovered recommendations: %#v", parsed.Recommendations)
}
}
func TestExtractCompleteRecommendationObjectsReturnsNilWhenArrayMissing(t *testing.T) {
if got := extractCompleteRecommendationObjects(`{"message":"no recommendations here"}`); len(got) != 0 {
t.Fatalf("expected no objects, got %#v", got)
}
}
func TestGeminiVisionMaxOutputTokensShrinksForSingleCandidate(t *testing.T) {
if got := geminiVisionMaxOutputTokens(1); got != 120 {
t.Fatalf("expected 120 tokens for single candidate, got %d", got)
}
if got := geminiVisionMaxOutputTokens(4); got != 300 {
t.Fatalf("expected 300 tokens for four candidates, got %d", got)
}
}
func TestParseSingleCandidateVisionTextParsesKeyValueResponse(t *testing.T) {
raw := "verdict: Yes\nassessment: positive\nrecommended: true\nreason_ko: 적합한 도시 야경\nsearch_hint: "
parsed, err := parseSingleCandidateVisionText(raw)
if err != nil {
t.Fatalf("expected parse success, got %v", err)
}
if parsed.Verdict != "Yes" || parsed.Assessment != "positive" || !parsed.Recommended {
t.Fatalf("unexpected parsed result: %#v", parsed)
}
if parsed.Reason != "적합한 도시 야경" {
t.Fatalf("unexpected reason: %#v", parsed)
}
}
func TestParseGeminiVisionLinesParsesPipeDelimitedRows(t *testing.T) {
raw := "0|Yes|positive|true|적합한 네온 도시|\n1|No|irrelevant|false|관련성 낮음|night city skyline"
parsed, ok := parseGeminiVisionLines(raw)
if !ok {
t.Fatal("expected pipe-delimited parser to succeed")
}
if len(parsed.Recommendations) != 2 {
t.Fatalf("unexpected parsed recommendations: %#v", parsed.Recommendations)
}
if parsed.Recommendations[0].Index != 0 || parsed.Recommendations[1].Index != 1 {
t.Fatalf("unexpected parsed indices: %#v", parsed.Recommendations)
}
}
+46 -15
View File
@@ -97,7 +97,7 @@ func RankSearchResults(query string, results []SearchResult) []SearchResult {
}
func GeminiCandidateLimit(total int) int {
return min(total, 16)
return min(total, 24)
}
func EvaluateAllCandidatesWithGemini(service *GeminiService, query string, ranked []SearchResult) ([]AIRecommendation, GeminiBatchStats, error) {
@@ -105,7 +105,7 @@ func EvaluateAllCandidatesWithGemini(service *GeminiService, query string, ranke
}
func EvaluateAllCandidatesWithGeminiWithDeadline(service *GeminiService, query string, ranked []SearchResult, deadline time.Time) ([]AIRecommendation, GeminiBatchStats, error) {
const chunkSize = 8
const chunkSize = 4
const maxConcurrentBatches = 2
if service == nil {
return nil, GeminiBatchStats{}, fmt.Errorf("gemini service is not configured")
@@ -186,7 +186,8 @@ func EvaluateAllCandidatesWithGeminiWithDeadline(service *GeminiService, query s
"error": batch.err.Error(),
})
}
recovered, recoveredErrs := recoverGeminiBatchSequentially(service, query, ranked, batch.index*chunkSize)
recovered, recoveredErrs := recoverGeminiBatchSequentially(service, query, ranked, batch.index*chunkSize, chunkSize, deadline)
hardErrs := filterHardGeminiErrors(recoveredErrs)
if len(recovered) > 0 {
stats.SequentialRetried++
stats.Succeeded++
@@ -197,14 +198,9 @@ func EvaluateAllCandidatesWithGeminiWithDeadline(service *GeminiService, query s
seen[item.Link] = true
merged = append(merged, item)
}
if len(recoveredErrs) > 0 {
stats.Failed++
for _, recoveredErr := range recoveredErrs {
if len(stats.Errors) < 5 {
stats.Errors = append(stats.Errors, recoveredErr)
}
}
}
continue
}
if len(hardErrs) == 0 {
continue
}
stats.Failed++
@@ -291,7 +287,7 @@ func ReviewedRecommendationLinks(items []AIRecommendation) map[string]bool {
}
func RemainingGeminiCapacity(reviewed []AIRecommendation) int {
remaining := GeminiCandidateLimit(16) - len(ReviewedRecommendationLinks(reviewed))
remaining := GeminiCandidateLimit(24) - len(ReviewedRecommendationLinks(reviewed))
if remaining < 0 {
return 0
}
@@ -345,11 +341,46 @@ func MergeGeminiBatchStats(base, extra GeminiBatchStats) GeminiBatchStats {
return merged
}
func recoverGeminiBatchSequentially(service *GeminiService, query string, ranked []SearchResult, startIndex int) ([]AIRecommendation, []string) {
recovered := make([]AIRecommendation, 0, 8)
func filterHardGeminiErrors(errs []string) []string {
filtered := make([]string, 0, len(errs))
for _, item := range errs {
if isIgnorableGeminiError(item) {
continue
}
filtered = append(filtered, item)
}
return filtered
}
func isIgnorableGeminiError(message string) bool {
lower := strings.ToLower(strings.TrimSpace(message))
if lower == "" {
return false
}
for _, token := range []string{
"no candidate thumbnails or preview frames could be fetched for gemini vision",
"candidate thumbnail is low value",
"candidate has no thumbnail or preview video",
"image url is empty",
} {
if strings.Contains(lower, token) {
return true
}
}
return false
}
func recoverGeminiBatchSequentially(service *GeminiService, query string, ranked []SearchResult, startIndex, chunkSize int, deadline time.Time) ([]AIRecommendation, []string) {
recovered := make([]AIRecommendation, 0, chunkSize)
errs := make([]string, 0, 4)
endIndex := min(startIndex+8, len(ranked))
endIndex := min(startIndex+chunkSize, len(ranked))
for idx := startIndex; idx < endIndex; idx++ {
if !deadline.IsZero() && time.Now().After(deadline) {
if len(errs) < 4 {
errs = append(errs, "sequential gemini recovery stopped at deadline")
}
break
}
recs, err := service.Recommend(query, []SearchResult{ranked[idx]})
if err != nil {
if len(errs) < 4 {
+3 -3
View File
@@ -15,7 +15,7 @@ type searchCollector interface {
type envatoCollector struct{}
func (envatoCollector) Name() string { return "Envato" }
func (envatoCollector) MaxResults() int { return 12 }
func (envatoCollector) MaxResults() int { return 16 }
func (envatoCollector) Enabled(enabledPlatforms map[string]bool) bool {
return len(enabledPlatforms) == 0 || enabledPlatforms["envato"]
}
@@ -31,7 +31,7 @@ func (envatoCollector) Enrich(searcher *SearchService, result SearchResult) Sear
type artgridCollector struct{}
func (artgridCollector) Name() string { return "Artgrid" }
func (artgridCollector) MaxResults() int { return 12 }
func (artgridCollector) MaxResults() int { return 16 }
func (artgridCollector) Enabled(enabledPlatforms map[string]bool) bool {
return len(enabledPlatforms) == 0 || enabledPlatforms["artgrid"]
}
@@ -47,7 +47,7 @@ func (artgridCollector) Enrich(searcher *SearchService, result SearchResult) Sea
type googleVideoCollector struct{}
func (googleVideoCollector) Name() string { return "Google Video" }
func (googleVideoCollector) MaxResults() int { return 8 }
func (googleVideoCollector) MaxResults() int { return 12 }
func (googleVideoCollector) Enabled(enabledPlatforms map[string]bool) bool {
return len(enabledPlatforms) == 0 || enabledPlatforms["google video"]
}
+26 -1
View File
@@ -38,6 +38,7 @@ const downloadLogs = document.getElementById("downloadLogs");
const debugLogList = document.getElementById("debugLogList");
const debugSummary = document.getElementById("debugSummary");
const resultModal = document.getElementById("resultModal");
const resultModalShell = document.getElementById("resultModalShell");
const resultModalTitle = document.getElementById("resultModalTitle");
const resultModalSource = document.getElementById("resultModalSource");
const resultModalSnippet = document.getElementById("resultModalSnippet");
@@ -56,6 +57,7 @@ const resultModalSecondaryAction = document.getElementById("resultModalSecondary
const closeResultModal = document.getElementById("closeResultModal");
const resultModalReady = Boolean(
resultModal &&
resultModalShell &&
resultModalTitle &&
resultModalSource &&
resultModalSnippet &&
@@ -436,12 +438,30 @@ function resetPreviewPlayer() {
function showModal(element) {
setHidden(element, false);
if (element === resultModal) {
syncResultModalLayout();
}
}
function hideModal(element) {
setHidden(element, true);
}
function syncResultModalLayout() {
if (!resultModalReady) {
return;
}
const viewportWidth = Math.max(320, window.innerWidth || document.documentElement.clientWidth || 0);
const viewportHeight = Math.max(320, window.innerHeight || document.documentElement.clientHeight || 0);
const shellWidth = Math.min(1150, viewportWidth - 16);
const shellHeight = Math.min(840, viewportHeight - 12);
const mediaHeight = Math.max(150, Math.min(Math.floor(viewportHeight * 0.32), 320));
resultModalShell.style.setProperty("--result-modal-shell-width", `${shellWidth}px`);
resultModalShell.style.setProperty("--result-modal-shell-height", `${shellHeight}px`);
resultModalShell.style.setProperty("--result-modal-media-max-height", `${mediaHeight}px`);
resultModalShell.classList.toggle("result-modal-compact", viewportHeight < 900 || viewportWidth < 1280);
}
function buildResultModalEmbedURL(item) {
if (item?.embedUrl) {
if (item.source === "Google Video" && item.embedUrl.includes("youtube-nocookie.com/embed/")) {
@@ -589,7 +609,7 @@ async function fetchResultPreview(item) {
}
const request = (async () => {
try {
const preview = await api("/api/download/preview", {
const preview = await api("/api/download/preview", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url: key }),
@@ -1030,6 +1050,11 @@ if (resultModalReady) {
}
});
}
window.addEventListener("resize", () => {
if (resultModalReady && !resultModal.classList.contains("hidden")) {
syncResultModalLayout();
}
});
for (const button of platformToggles) {
button.addEventListener("click", () => {
const platform = button.dataset.platformToggle;
+4 -4
View File
@@ -150,7 +150,7 @@
</div>
<div id="resultModal" class="fixed inset-0 z-50 hidden items-start justify-center overflow-y-auto bg-black/80 px-2 py-2 sm:px-4 sm:py-4">
<div class="result-modal-shell flex w-full max-w-6xl min-h-0 flex-col overflow-hidden rounded-3xl border border-white/10 bg-zinc-950 shadow-2xl">
<div id="resultModalShell" class="result-modal-shell flex w-full max-w-6xl min-h-0 flex-col overflow-hidden rounded-3xl border border-white/10 bg-zinc-950 shadow-2xl">
<div class="flex items-center justify-between border-b border-white/10 px-4 py-3 sm:px-5 sm:py-4">
<div class="min-w-0">
<p id="resultModalSource" class="text-xs uppercase tracking-[0.25em] text-zinc-500"></p>
@@ -180,13 +180,13 @@
</div>
</div>
<div class="result-modal-details grid min-h-0 gap-3 px-3 py-3 sm:gap-4 sm:px-4 sm:py-4 lg:grid-cols-[1.5fr_0.8fr]">
<div class="flex min-h-[180px] min-w-0 flex-col rounded-2xl border border-white/10 bg-white/[0.03] p-4">
<div class="flex min-h-0 min-w-0 flex-col rounded-2xl border border-white/10 bg-white/[0.03] p-4">
<p class="text-xs uppercase tracking-[0.25em] text-zinc-500">AI Note</p>
<div class="result-panel-scroll mt-3 min-h-0 flex-1 overflow-y-auto pr-2">
<p id="resultModalReason" class="whitespace-pre-wrap text-xs leading-6 text-zinc-200 sm:text-sm"></p>
</div>
</div>
<div class="flex min-h-[200px] min-w-0 flex-col rounded-2xl border border-white/10 bg-white/[0.03] p-4">
<div class="flex min-h-0 min-w-0 flex-col rounded-2xl border border-white/10 bg-white/[0.03] p-4">
<div class="mb-3 flex flex-col gap-2">
<button id="resultModalDownload" type="button" class="hidden w-full rounded-2xl border border-white bg-white px-4 py-3 text-sm font-medium text-black transition hover:bg-zinc-200">
Primary Action
@@ -224,6 +224,6 @@
</button>
</template>
<script src="/app.js?v=20260317c" defer></script>
<script src="/app.js?v=20260317d" defer></script>
</body>
</html>
+13 -2
View File
@@ -60,7 +60,8 @@ body {
}
.result-modal-shell {
height: min(calc(100dvh - 0.5rem), 860px);
height: var(--result-modal-shell-height, min(calc(100dvh - 0.5rem), 860px));
max-width: var(--result-modal-shell-width, 72rem);
margin: auto;
}
@@ -69,7 +70,7 @@ body {
}
.result-modal-media-frame {
max-height: min(34dvh, 22rem);
max-height: var(--result-modal-media-max-height, min(34dvh, 22rem));
}
.result-modal-details {
@@ -79,6 +80,16 @@ body {
align-items: stretch;
}
.result-modal-shell.result-modal-compact .result-modal-details {
gap: 0.5rem;
}
.result-modal-shell.result-modal-compact #resultModalReason,
.result-modal-shell.result-modal-compact #resultModalSnippet {
font-size: 0.75rem;
line-height: 1.35rem;
}
#resultModalSnippet,
#resultModalReason {
white-space: pre-wrap;
+280
View File
@@ -0,0 +1,280 @@
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$script:RepoRoot = Split-Path -Parent $PSScriptRoot
$script:ToolsRoot = Join-Path $script:RepoRoot ".tools"
$script:BinRoot = Join-Path $script:ToolsRoot "bin"
$script:GoRoot = Join-Path $script:ToolsRoot "go"
$script:GoBin = Join-Path $script:GoRoot "bin"
$script:FFmpegRoot = Join-Path $script:ToolsRoot "ffmpeg"
$script:FFmpegBin = Join-Path $script:FFmpegRoot "bin"
$script:VenvRoot = Join-Path $script:RepoRoot ".venv"
$script:VenvScripts = Join-Path $script:VenvRoot "Scripts"
$script:LocalRoot = Join-Path $script:RepoRoot ".local"
$script:GoPathRoot = Join-Path $script:LocalRoot "go"
$script:GoModCacheRoot = Join-Path $script:GoPathRoot "pkg\mod"
$script:GoBuildCacheRoot = Join-Path $script:LocalRoot "go-build-cache"
$script:CredentialFile = Join-Path $script:LocalRoot "git-credentials.psd1"
function Write-Step {
param([string]$Message)
Write-Host "[ai-media-hub] $Message"
}
function Invoke-CheckedCommand {
param(
[Parameter(Mandatory = $true)][string]$FilePath,
[string[]]$Arguments = @()
)
& $FilePath @Arguments
if ($LASTEXITCODE -ne 0) {
$joined = if ($Arguments.Count -gt 0) { " " + ($Arguments -join " ") } else { "" }
throw "명령 실행 실패: $FilePath$joined"
}
}
function Ensure-Directory {
param([Parameter(Mandatory = $true)][string]$Path)
if (-not (Test-Path -LiteralPath $Path)) {
New-Item -ItemType Directory -Path $Path | Out-Null
}
}
function Get-RepoRoot { return $script:RepoRoot }
function Get-ToolsRoot { return $script:ToolsRoot }
function Get-BinRoot { return $script:BinRoot }
function Get-GoRoot { return $script:GoRoot }
function Get-GoBin { return $script:GoBin }
function Get-FFmpegBin { return $script:FFmpegBin }
function Get-VenvRoot { return $script:VenvRoot }
function Get-VenvScripts { return $script:VenvScripts }
function Get-LocalRoot { return $script:LocalRoot }
function Get-GoPathRoot { return $script:GoPathRoot }
function Get-GoModCacheRoot { return $script:GoModCacheRoot }
function Get-GoBuildCacheRoot { return $script:GoBuildCacheRoot }
function Get-CredentialFile { return $script:CredentialFile }
function Resolve-SystemCommand {
param([Parameter(Mandatory = $true)][string]$Name)
try {
return (Get-Command $Name -ErrorAction Stop).Source
} catch {
return $null
}
}
function Resolve-PythonExe {
$candidates = @(
(Join-Path $script:VenvScripts "python.exe"),
(Resolve-SystemCommand -Name "python")
)
foreach ($candidate in $candidates) {
if ($candidate -and (Test-Path -LiteralPath $candidate)) {
return $candidate
}
}
throw "python.exe를 찾을 수 없습니다. Python 3.12+가 필요합니다."
}
function Resolve-BasePythonExe {
$all = @(Get-Command python -All -ErrorAction SilentlyContinue)
foreach ($item in $all) {
if ($item.Source -and $item.Source -notlike "*\.venv\*") {
return $item.Source
}
}
$fallback = Resolve-SystemCommand -Name "python"
if ($fallback -and $fallback -notlike "*\.venv\*") {
return $fallback
}
throw "시스템 python.exe를 찾지 못했습니다."
}
function Resolve-VenvPythonExe {
$pythonExe = Join-Path $script:VenvScripts "python.exe"
if (-not (Test-Path -LiteralPath $pythonExe)) {
throw ".venv가 아직 준비되지 않았습니다. scripts/setup-dev.ps1를 먼저 실행하세요."
}
return $pythonExe
}
function Use-LocalTooling {
Ensure-Directory -Path $script:ToolsRoot
Ensure-Directory -Path $script:BinRoot
Ensure-Directory -Path $script:LocalRoot
Ensure-Directory -Path $script:GoPathRoot
Ensure-Directory -Path $script:GoModCacheRoot
Ensure-Directory -Path $script:GoBuildCacheRoot
$segments = @(
$script:BinRoot,
$script:GoBin,
$script:FFmpegBin,
$script:VenvScripts,
$env:PATH
) | Where-Object { $_ -and $_.Trim() -ne "" }
$env:PATH = ($segments | Select-Object -Unique) -join ";"
$env:GOPATH = $script:GoPathRoot
$env:GOMODCACHE = $script:GoModCacheRoot
$env:GOCACHE = $script:GoBuildCacheRoot
}
function Ensure-Venv {
$venvPython = Join-Path $script:VenvScripts "python.exe"
if (Test-Path -LiteralPath $venvPython) {
return $venvPython
}
Ensure-Directory -Path $script:LocalRoot
$pythonExe = Resolve-SystemCommand -Name "python"
if (-not $pythonExe) {
throw "시스템 python이 없어 .venv를 만들 수 없습니다."
}
Write-Step ".venv 생성"
& $pythonExe -m venv $script:VenvRoot
return $venvPython
}
function Install-PythonRequirements {
param([switch]$UpgradePip)
$venvPython = Ensure-Venv
$systemPython = Resolve-BasePythonExe
if ($UpgradePip) {
Write-Step "pip 업그레이드"
Invoke-CheckedCommand -FilePath $systemPython -Arguments @("-m", "pip", "--python", $venvPython, "install", "--upgrade", "pip")
}
Write-Step "worker requirements 설치"
Invoke-CheckedCommand -FilePath $systemPython -Arguments @("-m", "pip", "--python", $venvPython, "install", "-r", (Join-Path $script:RepoRoot "worker\requirements.txt"))
}
function New-CmdShim {
param(
[Parameter(Mandatory = $true)][string]$Path,
[Parameter(Mandatory = $true)][string]$Target
)
$content = @(
"@echo off",
"set SCRIPT_DIR=%~dp0",
"`"%SCRIPT_DIR%$Target`" %*"
) -join "`r`n"
Set-Content -LiteralPath $Path -Value $content -Encoding ASCII
}
function Ensure-CommandShims {
Ensure-Directory -Path $script:BinRoot
$pythonShimTarget = "..\..\ .venv\Scripts\python.exe".Replace(" ", "")
New-CmdShim -Path (Join-Path $script:BinRoot "python3.cmd") -Target $pythonShimTarget
$ytDlpExe = Join-Path $script:VenvScripts "yt-dlp.exe"
if (Test-Path -LiteralPath $ytDlpExe) {
New-CmdShim -Path (Join-Path $script:BinRoot "yt-dlp.cmd") -Target "..\..\ .venv\Scripts\yt-dlp.exe".Replace(" ", "")
}
}
function Test-GoReady {
return [bool](Resolve-SystemCommand -Name "go")
}
function Test-FFmpegReady {
return [bool](Resolve-SystemCommand -Name "ffmpeg")
}
function Test-YtDlpReady {
return [bool](Resolve-SystemCommand -Name "yt-dlp")
}
function Install-LocalGo {
param(
[string]$Version = "1.24.0",
[string]$ArchiveUrl = ""
)
if (Test-Path -LiteralPath (Join-Path $script:GoBin "go.exe")) {
Write-Step "로컬 Go 이미 준비됨"
return
}
if (-not $ArchiveUrl) {
$ArchiveUrl = "https://go.dev/dl/go$Version.windows-amd64.zip"
}
Ensure-Directory -Path $script:ToolsRoot
$archivePath = Join-Path $script:LocalRoot "go-$Version.windows-amd64.zip"
Write-Step "로컬 Go 다운로드: $ArchiveUrl"
Invoke-WebRequest -Uri $ArchiveUrl -OutFile $archivePath
if (Test-Path -LiteralPath $script:GoRoot) {
Remove-Item -Recurse -Force $script:GoRoot
}
Expand-Archive -LiteralPath $archivePath -DestinationPath $script:ToolsRoot -Force
}
function Install-LocalFFmpeg {
param([string]$ArchiveUrl = "https://www.gyan.dev/ffmpeg/builds/ffmpeg-release-essentials.zip")
if (Test-Path -LiteralPath (Join-Path $script:FFmpegBin "ffmpeg.exe")) {
Write-Step "로컬 ffmpeg 이미 준비됨"
return
}
Ensure-Directory -Path $script:ToolsRoot
$archivePath = Join-Path $script:LocalRoot "ffmpeg-release-essentials.zip"
Write-Step "로컬 ffmpeg 다운로드: $ArchiveUrl"
Invoke-WebRequest -Uri $ArchiveUrl -OutFile $archivePath
$extractRoot = Join-Path $script:LocalRoot "ffmpeg-extract"
if (Test-Path -LiteralPath $extractRoot) {
Remove-Item -Recurse -Force $extractRoot
}
Expand-Archive -LiteralPath $archivePath -DestinationPath $extractRoot -Force
$binCandidate = Get-ChildItem -Path $extractRoot -Recurse -Filter "ffmpeg.exe" | Select-Object -First 1
if (-not $binCandidate) {
throw "ffmpeg.exe를 압축 파일에서 찾지 못했습니다."
}
$sourceBin = Split-Path -Parent $binCandidate.FullName
if (Test-Path -LiteralPath $script:FFmpegRoot) {
Remove-Item -Recurse -Force $script:FFmpegRoot
}
Ensure-Directory -Path $script:FFmpegBin
Copy-Item -Path (Join-Path $sourceBin "*") -Destination $script:FFmpegBin -Recurse -Force
}
function Get-AppEnvironment {
param(
[Parameter(Mandatory = $true)][string]$WorkspaceRoot,
[Parameter(Mandatory = $true)][string]$DownloadsDir,
[Parameter(Mandatory = $true)][string]$SqlitePath,
[Parameter(Mandatory = $true)][string]$AppAddr,
[string]$SearxUrl = "http://127.0.0.1:18080"
)
return @{
APP_ROOT = $WorkspaceRoot
APP_ADDR = $AppAddr
SQLITE_PATH = $SqlitePath
DOWNLOADS_DIR = $DownloadsDir
FRONTEND_DIR = (Join-Path $WorkspaceRoot "frontend")
WORKER_SCRIPT = (Join-Path $WorkspaceRoot "worker\downloader.py")
SEARXNG_BASE_URL = $SearxUrl
SEARXNG_GOOGLE_VIDEO_ENGINE = "google videos"
SEARXNG_WEB_ENGINE = "google"
}
}
function Import-GitCredential {
$credentialFile = Get-CredentialFile
if (-not (Test-Path -LiteralPath $credentialFile)) {
throw "git 자격정보 파일이 없습니다: $credentialFile"
}
return Import-PowerShellDataFile -Path $credentialFile
}
+10
View File
@@ -0,0 +1,10 @@
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
Write-Step "로컬 개발 PATH 적용 완료"
Write-Host (" Repo : " + (Get-RepoRoot))
Write-Host (" .venv : " + (Get-VenvRoot))
Write-Host (" .tools : " + (Get-ToolsRoot))
+31
View File
@@ -0,0 +1,31 @@
param(
[string]$Remote = "origin",
[string]$Branch = "",
[switch]$SetUpstream
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
$credential = Import-GitCredential
if (-not $Branch) {
$Branch = (& git branch --show-current).Trim()
}
if (-not $Branch) {
throw "현재 브랜치를 찾지 못했습니다."
}
$pair = "{0}:{1}" -f $credential.Username, $credential.Password
$token = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($pair))
$gitArgs = @("-c", "http.extraheader=AUTHORIZATION: Basic $token", "push")
if ($SetUpstream) {
$gitArgs += @("--set-upstream", $Remote, $Branch)
} else {
$gitArgs += @($Remote, $Branch)
}
Write-Step "git push $Remote $Branch"
& git @gitArgs
+28
View File
@@ -0,0 +1,28 @@
param(
[string]$AppAddr = "127.0.0.1:8080",
[string]$SqlitePath = "",
[string]$DownloadsDir = "",
[string]$SearxUrl = "http://127.0.0.1:18080"
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
$repoRoot = Get-RepoRoot
if (-not $SqlitePath) {
$SqlitePath = Join-Path $repoRoot "db\media.dev.db"
}
if (-not $DownloadsDir) {
$DownloadsDir = Join-Path $repoRoot "downloads"
}
$envMap = Get-AppEnvironment -WorkspaceRoot $repoRoot -DownloadsDir $DownloadsDir -SqlitePath $SqlitePath -AppAddr $AppAddr -SearxUrl $SearxUrl
foreach ($entry in $envMap.GetEnumerator()) {
Set-Item -Path ("Env:" + $entry.Key) -Value $entry.Value
}
Write-Step "백엔드 실행"
Invoke-CheckedCommand -FilePath "go" -Arguments @("run", "./backend")
+158
View File
@@ -0,0 +1,158 @@
param(
[int]$MockPort = 18080,
[int]$AppPort = 18081,
[switch]$SkipGoFmt
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
$repoRoot = Get-RepoRoot
$tmpRoot = Join-Path (Get-LocalRoot) ("selftest-" + [guid]::NewGuid().ToString("N"))
Ensure-Directory -Path $tmpRoot
$mockProcess = $null
$appProcess = $null
$appStdoutTask = $null
$appStderrTask = $null
function Stop-TrackedProcess {
param($Process)
if ($null -ne $Process -and -not $Process.HasExited) {
Stop-Process -Id $Process.Id -Force
$Process.WaitForExit()
}
}
try {
if (-not $SkipGoFmt) {
Write-Step "gofmt"
Invoke-CheckedCommand -FilePath "gofmt" -Arguments @(
"-w",
(Join-Path $repoRoot "backend\main.go"),
(Join-Path $repoRoot "backend\handlers\api.go"),
(Join-Path $repoRoot "backend\models\db.go"),
(Join-Path $repoRoot "backend\services\cse.go"),
(Join-Path $repoRoot "backend\services\cse_test.go"),
(Join-Path $repoRoot "backend\services\search_collectors.go"),
(Join-Path $repoRoot "backend\services\ranker.go"),
(Join-Path $repoRoot "backend\services\gemini.go"),
(Join-Path $repoRoot "backend\services\gemini_test.go")
)
}
Write-Step "python syntax"
Invoke-CheckedCommand -FilePath (Resolve-VenvPythonExe) -Arguments @(
"-m",
"py_compile",
(Join-Path $repoRoot "worker\downloader.py"),
(Join-Path $repoRoot "scripts\mock_searxng.py")
)
Write-Step "go test"
Invoke-CheckedCommand -FilePath "go" -Arguments @("test", "./...")
Write-Step "frontend syntax"
Invoke-CheckedCommand -FilePath "node" -Arguments @("--check", (Join-Path $repoRoot "frontend\app.js"))
Write-Step "go build"
$binaryPath = Join-Path $tmpRoot "ai-media-hub.exe"
Invoke-CheckedCommand -FilePath "go" -Arguments @("build", "-o", $binaryPath, "./backend")
Write-Step "start mock searxng"
$mockLog = Join-Path $tmpRoot "mock-searxng.stdout.log"
$mockErrLog = Join-Path $tmpRoot "mock-searxng.stderr.log"
$mockProcess = Start-Process -FilePath (Resolve-VenvPythonExe) `
-ArgumentList @((Join-Path $repoRoot "scripts\mock_searxng.py"), "--port", $MockPort) `
-RedirectStandardOutput $mockLog `
-RedirectStandardError $mockErrLog `
-PassThru `
-WindowStyle Hidden
Write-Step "start app"
$downloadsDir = Join-Path $tmpRoot "downloads"
Ensure-Directory -Path $downloadsDir
$appLog = Join-Path $tmpRoot "app.log"
$envMap = Get-AppEnvironment `
-WorkspaceRoot $repoRoot `
-DownloadsDir $downloadsDir `
-SqlitePath (Join-Path $tmpRoot "media.db") `
-AppAddr ("127.0.0.1:{0}" -f $AppPort) `
-SearxUrl ("http://127.0.0.1:{0}" -f $MockPort)
$appStartInfo = New-Object System.Diagnostics.ProcessStartInfo
$appStartInfo.FileName = $binaryPath
$appStartInfo.WorkingDirectory = $repoRoot
$appStartInfo.UseShellExecute = $false
$appStartInfo.RedirectStandardOutput = $true
$appStartInfo.RedirectStandardError = $true
foreach ($entry in $envMap.GetEnumerator()) {
$appStartInfo.Environment[$entry.Key] = [string]$entry.Value
}
$appProcess = New-Object System.Diagnostics.Process
$appProcess.StartInfo = $appStartInfo
$null = $appProcess.Start()
$appStdoutTask = $appProcess.StandardOutput.ReadToEndAsync()
$appStderrTask = $appProcess.StandardError.ReadToEndAsync()
$healthUrl = "http://127.0.0.1:$AppPort/healthz"
$healthy = $false
for ($i = 0; $i -lt 30; $i++) {
try {
$health = Invoke-RestMethod -Uri $healthUrl -TimeoutSec 2
if ($health.status -eq "ok") {
$healthy = $true
break
}
} catch {
Start-Sleep -Seconds 1
}
}
if (-not $healthy) {
throw "/healthz 확인 실패"
}
Write-Step "verify search"
$searchPayload = @{
query = "city rain"
platforms = @("envato", "artgrid", "google video")
} | ConvertTo-Json
$search = Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:$AppPort/api/search" -ContentType "application/json" -Body $searchPayload
$searchResults = @($search.results)
if ($null -eq $search.results -or $searchResults.Count -lt 2) {
throw "검색 결과가 너무 적습니다."
}
if (@($searchResults | Where-Object { -not $_.link }).Count -gt 0) {
throw "검색 결과에 link가 비어 있습니다."
}
Write-Step "verify upload"
$sampleFile = Join-Path $tmpRoot "sample.txt"
Set-Content -LiteralPath $sampleFile -Value "selftest upload" -Encoding UTF8
$form = @{
file = Get-Item -LiteralPath $sampleFile
}
$upload = Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:$AppPort/api/upload" -Form $form
if (-not $upload.filename) {
throw "업로드 응답에 filename이 없습니다."
}
$uploadedPath = Join-Path $downloadsDir $upload.filename
if (-not (Test-Path -LiteralPath $uploadedPath)) {
throw "업로드된 파일이 downloads에 존재하지 않습니다."
}
Write-Step "selftest ok"
} finally {
Stop-TrackedProcess -Process $appProcess
Stop-TrackedProcess -Process $mockProcess
if ($appStdoutTask) {
$appStdoutTask.Wait()
$appStdoutTask.Result | Set-Content -LiteralPath (Join-Path $tmpRoot "app.log") -Encoding UTF8
}
if ($appStderrTask) {
$appStderrTask.Wait()
$appStderrTask.Result | Add-Content -LiteralPath (Join-Path $tmpRoot "app.log") -Encoding UTF8
}
}
+43
View File
@@ -0,0 +1,43 @@
param(
[switch]$SkipPythonDeps,
[switch]$SkipGoDownload,
[switch]$SkipFFmpegDownload,
[switch]$UpgradePip
)
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
. (Join-Path $PSScriptRoot "dev-tools.ps1")
Use-LocalTooling
Ensure-Directory -Path (Get-ToolsRoot)
Ensure-Directory -Path (Get-LocalRoot)
$pythonExe = Ensure-Venv
Write-Step "Python 사용 경로: $pythonExe"
if (-not $SkipPythonDeps) {
Install-PythonRequirements -UpgradePip:$UpgradePip
}
Ensure-CommandShims
if (-not $SkipGoDownload -and -not (Test-GoReady)) {
Install-LocalGo
}
if (-not $SkipFFmpegDownload -and -not (Test-FFmpegReady)) {
Install-LocalFFmpeg
}
Use-LocalTooling
Write-Step "준비 상태"
Write-Host (" python3: " + ((Resolve-SystemCommand -Name "python3") ?? "미설치"))
Write-Host (" python : " + ((Resolve-SystemCommand -Name "python") ?? "미설치"))
Write-Host (" yt-dlp : " + ((Resolve-SystemCommand -Name "yt-dlp") ?? "미설치"))
Write-Host (" go : " + ((Resolve-SystemCommand -Name "go") ?? "미설치"))
Write-Host (" ffmpeg : " + ((Resolve-SystemCommand -Name "ffmpeg") ?? "미설치"))
Write-Step "현재 셸에 PATH를 적용하려면 다음처럼 dot-source 하세요."
Write-Host " . .\scripts\enter-dev-shell.ps1"