Bifrost - Coding Agents on Mobile

Speaker 0: Good afternoon. Alright. Speaker 1: Good afternoon, everyone. My name is Ebraheem Alrawi, a student chat Amity University Dubai. And Day, I want to share with you my major project for, my university course. This is Speak mk1, a mamba-attention model from scratch and all local, and this is specifically for speech therapy. Now there are 2 main things I wanted to target with this model. Speaker 1: The fact that there is a like there is a lack of multilingual, speech language Scene speech language pathologists, especially in Speaker 2: The Speaker 1: UAE. And, because it's targeted for children, children can AI, like, the the attention span, their attention span can be, AI, their attendees span can be LLM. Like AI, 5 seconds in, they are distracted with something else. And APIs, and this aims to be as accurate and as fast as possible to reach to the child. Now, I'm gonna run the API over here first off. Speaker 1: Okay. It's running. Alright. Now I'm gonna input a record a videos, audio recording of a child with a speech impediment, and this is the text. So I'm now gonna send this to, the LLM and tool the multimodal Ady, to the multimodal model that features the audio encoder and the LLM. Speaker 1: And if you give it a few more seconds yeah. There use go. Now it may be Ajay. Now it may be garbled, but notices what the impediment what the impediment is, and it tries to fix the impediment for the child. So, like, instead of sharing, like, it tells it tells a child to say in, another technique. Speaker 1: Right? Now, and this is also wait. Hold on. AI just gonna Speaker 3: For 1 second. Let's practice this 1. Let's use air to enable work. Means s s s s s s s Speaker 1: So, yeah, this is the, multimodal model. It's It's tool, work in progress. Now, I want to share with you, like, the development stage from, where from APIs, from the audio encoder to the multimodal. Now, the audio encoder goes through a a Gemma encode a Mamba encoder Zustand then to the local window, window attendees. And then it branches to both the bidirectional Mamba and the CG MLP. Speaker 1: And afterwards, chat will merge it together to form an Coding of the audio. Now we've Mhate train this on LibriSpeech with multimodal. With the Montreal Forced Aligner so that it can extract the phonemes out of it. And the best epoch that we've got was epoch 13. Most of the stats are in the nineties with overall accuracy of 82.1. Speaker 1: The place is 40%, but this is because of the log file spectrum having issues of finding the place of the of the mod of, the audio. But as long as the author, other, like, status out there Armaan the nineties, it should it it will work just as fine. Now, if we go to the LLM here, the LLM is, a 74,000,000 parameter model, tokenized using GPT Neo x. We first train so, yeah, this is also a Mamba-attention hybrid, decoder. Now we've trained this on, now we first train this on like Armaan AI stories. Speaker 1: Right? And, specifically, the g p t 4 Scene by Campus. Carpati. If I can show you the the, model for the output for the first stage of the model. Now I'll just type in a prompt over here, and the s and the, model will type in, the prompt app you can see down here. Speaker 1: And, yeah, this is the prompt from AI stories. Now beforehand 1 second. The previous model had issues with there was a previous issue where I had a bubbles where I had a double shift. I teaching, some people saw the image of, like, the first model, the, the first output of the model on Courts HQ. If I could show you where it where is it, though? Speaker 4: Where is the histories? Train share. No. Speaker 1: Yeah. I can't seem to find it. But, yeah, it it had Arabic of stutters. It said a few words that didn't make Scene. And that and the issue was because of the double shift that I've double shift I've had. Speaker 1: Where would Speaker 4: chat is Speaker 1: ah, right over here. This part was the cause of the double shift. And, once I've removed City, once I've, commented this part out, I've had this, output right over here app you can see down there. Now, we've trained this all the way through stage 4. We've, put childless data onto City, and we've integrated with PubMed Central's speech therapy Day speech therapy papers. Speaker 1: And then finally, we, I've made a synthetic data chat specifically meant for SLP AI, and the and this was generated AI, via Gemma 3 n. Speaker 4: The Speaker 1: and the problem was when we AI to integrate this to a multimodal, solution, The problem was that the audio Cloud not go through the LLM. And as a result, it reducing, a garbled results that were from the audiobook that were from the Speech Day. And as a temporary solution, forced to open the gates, at the cross attention, Sponsors, no. Sorry. CrossModalSparseAttention, which temporarily lead the issue right over here. Speaker 1: So now inputted the SLP dialogue. This is, after all, a work in progress model. On future training models on future models, I'm going tool, input more, child, like, AI, implement audio, datasets onto the audio encoder and also integrate audio with the text language so that it can better so that the language model can better understand the, context. And, yeah, this is my Speak 1 project. This is my GitHub. Speaker 1: If right here, you can start the project over here. And, yeah, I'm now open to any questions. Thank you. Speaker 5: You mentioned you mentioned you Speaker 4: know that they did this speech. Speaker 5: Yeah. But, that has a lot of noise in BITS, like a lot of call centers, talking, background noise, music also. So, would I I and that's 1 1 thing. I'm wondering now what better Day audio dataset you could have used. Speaker 1: So first off, I've used the ultra the clean version of the Day, so that's the 1 without the noise and stuff. The second thing, I'm planning to use the UltraSuite Day, more specifically, the UX TD and UX 20 20. That's what I'm planning to train the audio encoder on. And for the, language model, AI probably gonna modify the synthetic Scene, it's Pydantic dataset so that, I integrate the author with the, dialogue stuff. So yeah. Speaker 5: And and what type of speech impediments? Because there are a few. Can this model detect right now? Speaker 1: Right now, it detects articulations articulation errors. So instead of r, it'll be force. Lead of Institute tool, like, or it it doesn't pronounce the s correct. It doesn't pronounce the s. So instead of saying yesterday, it'll be author. Speaker 5: Yeah. Oh, okay. Thank you very much. Speaker 1: Thank you. Speaker 5: Thank you so much. Speaker 3: Did you build the model by yourself? Speaker 1: Yes. Speaker 3: How? Speaker 1: I did get some assistance for I did have, like, some background in, like, trying to make my own models from scratch. That was a practice. But then I also Hadi AI to assist me through my journey as well. Like. Because this is, after all, a newer Courts of, model. Speaker 1: There's not much research into Mamba as much as transformers. So this is a so, yeah, most of it is, like, AI and my own experience. Speaker 3: Very cool. Does this only support English for now? Does this only support English for now? Speaker 1: Yeah. The oh, for now, it only supports English. But in future releases role-shaped, it will be training Arabic, maybe AI Mandarin force and Japanese. Speaker 3: Nice. Thank you. Speaker 2: Thank you. Speaker 4: Ngom problem. Anyone else? Yeah. That Yep. Okay. Speaker 4: I think AI, the next time, finally, we, Speaker 5: Sure. Speaker 3: Yes. Thank you for confirming. Speaker 4: No. No. Speaker 2: What do Speaker 5: you think? What do you think? Speaker 2: AI? Okay. Speaker 4: Not the other way around. The small piece is the But the system line? That's the audio. K? You can put it here. Speaker 4: Hello? Yes. That's all good. Just mute. Just stay on mute. Speaker 2: Okay. I'm Speaker 4: just I just zoom in on the page. But I know. Mhmm. That's 1 thing which gets very difficult to see from But Speaker 2: I don't know. AI. I'll so I have to swap between phone and laptop because it's on the thing. Hardware Speaker 4: recording over here, but as long as the people Author. Speaker 2: Okay. Right. Okay. Cool. Hey, everyone. Speaker 2: I'm Ajay. I'm here to demo Bifrost. So, I run app product studio, which means we're developing software for different people, companies, and so on. And AI everybody else in this room, we use coding agents for our Software. Right? Speaker 2: Predominantly Cloud Code and, Correct. Now the thing is these are great tools, but they require you to be at your app, right, because you have your files on local, common, so forth. And I wanted an easy way of being able to do all of this coding work, run these agents, etcetera, through my phone. Now some native solutions Artist, both Cloud Code and ODeX expose remote Context, therapy Cloud products Zustand so on. But, like, you know, remote of those isn't, like, uniform Zustand I, like, kinda wanted to build my own tool anyway, and that's Bifrost. Speaker 2: And Venue the name. Right? Bifrost is a bridge in Norse mythology, and this is a bridge between my phone here and my laptop. So what this is doing is I have Bifrost running locally on my machine right now. It's exposing this front end and back end, and I can I, whenever I send a message, it will spawn a Cloud code or a codex agent on my laptop, which can do whatever I wanted to do? Speaker 2: So we'll start with something simple. I'll say, called AI Tinkerers. Right? App. So this you can see that, because AI sent a message, I have this log thing going on, and you can see the conversation I'm having with, Cloud as well. Speaker 2: So now this is spawning a cloud code instance on my laptop, which has access to all my files, of course, because it's running on my laptop and it can do whatever I lead it to do. And it works the same way. Right? Like, okay, through, you know, there is a tool result error as well, and it's saying AI something something. Let me just keep going down. Speaker 2: Okay. And then AI, it says the I'm sorry. I don't know if you can see. Okay. It says that the folder exists. Speaker 2: Let's check it out. Lead. AI Tinkerers costs. Obviously, this is, like, the most basic lightweight thing. Right? Speaker 2: Like but you can keep going AI it. No? The complexity is, like, arbitrary, because it's the exact same correct or Cloud code with the same tools, hooks, all the capabilities as if it were running on your laptop. Right? Let me say free a simple timer app in this folder. Speaker 4: It should be stream. Speaker 2: Yeah. I'll just scroll. So you can see the last. And, like, you know, this is, like, streaming all the tool scale. It's thinking so on so forth Custom as it would normally on the terminals, right, if you were using a coding agent on your laptop. Speaker 2: Okay. It says file created successfully. There it is. Ajay. Of Courts, I didn't give it okay. Speaker 2: And it thankfully, it works too. But, yeah, this is the, you know, this is the basic stuff. Some other features that I've built in is because if you free working on something and then you have to leave, you want the ability to continue a conversation that you already started. Right? So we have this feature called takeover, which will load all the local sessions, all the local chat that you have already training, and you'll be able to, you know, take them over from here and continue. Speaker 2: And and it goes back and Courts you can oh, sorry. So we have this feature called takeover, yeah, which shows you all the existing conversations that you have on your app. And you can so you can swap between your phone and, laptop free, you know, start on the phone, continue on the machine, start on the machine, continue on the phone, and so on. And this extends to arbitrary complexity. Right? Speaker 2: Let's say I let me pick something. Or if I you know, it chat also run files and, like, access repos, give you information on, like, BITS history, so on so forth. So, yeah, essentially, Talwar all this does is it's opening up a terminal on my, laptop. It's invoking ODeX or Cloud, and it's managing chat between my phone and my laptop. That's it. Speaker 2: All the capabilities, everything on the, are by the agents themselves. Yep. Yeah. So, yeah, you can see the conversation going on and, you know, it just continues. I I think I'll stop doing the demo now. Speaker 2: This this can be extended to teaching. Right? Like, more complex programming projects, simpler stuff, whatever it is, and talk a little bit about how how this works underneath the hood. So now the train, LLM me see. Okay. Speaker 2: So the main Courts of building block of this entire thing is this. Is it not oh, okay. Okay. Alright? Cool. Speaker 2: Ajay. So now the core building block of this entire tool is this thing called the ODeX exec loop. So normally Menu you launch codecs, you have something like this Coding. Right? Like, where, you know, you're teaching. Speaker 2: It's a 2 way conversation. I've sent it a message saying start AI. It's doing x y z, and then I can respond to it. What we do what I've done here is instead of doing that, I've used what is this, this Stack loop. What that does is it gives you a single turn conversation. Speaker 2: So you send 1 message, the agent gives you back 1 response, and then the conversation ends. So, we do something AI, just to give you an Campus. So you can see that it's, like, you know, doing all of this. The model is, ODeX and Cloud code both expose commands that let you choose the model and so on. Okay. Speaker 2: Oh, wow. I said pick a color between red and blue app red and purple. But anyway, so this is how you spawn sessions on your, this is how it Speak sessions on my laptop. And now the core teaching, what makes all of this work is mid-session ID. Right? Speaker 2: So if I just look for the session ID locally, you'll see that, Cloud Code and Codefs, they both give us the full history of the conversation in these JSON files. So my question will be here right between red and blue, and it'll have, like, you know, BITS response and so on to tool scale, if any. And these agents scale also let you stream all of this data, which is how when you in the mobile demo, you saw that as ODeX was thinking, output was coming and you could see the tool calls being used, thinking and so on. But essentially the entire thing that this app does is that it spawns a single ODeX exec, command. It gets a response. Speaker 2: It stores this session ID. And when you app a user send the next message, it just because it already has the session ID stored in memory, it takes that and continues the Conversation. And this just keeps going on for HINTS steps. So so this basic exec is the is the sort of core workhorse of how this entire thing works. I'll quickly go to the code and show you a couple of files which are interesting. Speaker 2: So I have this, Bifrost repo, front end, back Engine both. Front end is fairly straightforward. It's just a Real-time Date app that opens a WebSocket for chat. Nothing major. Most of the work happens in the back end. Speaker 2: Please train. Okay. So what is interesting is this agent Realtime registry because this is where all the heavy lifting of spawning a cloud session, spawning a codec session, and so on happens. Yeah. So you see this, build Cloud AI, it's creating a Mode Day runtime with, certain commands for cloud. Speaker 2: Right? So it says exec resume. Prompt is what we input. This JSON tells Cloud and ODeX how to give the output output as Day, and you'll see this, slightly risky command where I've said, don't ask me for approval for tool calls. You just, you know, do whatever you want. Speaker 2: But, of course, this is all, like, customizable. Right? You can give these agents as much power or as little as you want. So so this agent AI registry is where most of this actual connection settings, sending the command from your phone to laptop Zustand so on happens. The other interesting part of this is, how we, I showed you this JSON blob that, the agents give us, right, which has everything. Speaker 2: How we force it in order to, give it as a proper output on the phone. So, you build see here that, you know, we'll be sorry. Parsing tool calls. Yeah. You see tool name, tool use Zustand so on. Speaker 2: So if you go back to the yeah. So so this what what it does is it takes that JSON app, the agent is streaming out that JSON. It picks all of these different, what do you call like values from the JSON itself. Right? What is the tool? Speaker 2: What is the thought process? What is the user's message? What is the agent's message Zustand so on? And then, formats in a way such that it can clearly be shown on a mobile phone. So so that's the front end and back end. Speaker 2: Now another sort of important part of this is the security aspect. Right? Because, anyone who can access who has access to this agent now has access to my machine. So all my Menu files and so on, everything. And this was a common problem with OpenAI. Speaker 2: Right? Because, the, a straightforward way to do this is to just expose APIs. You'll be able to see this here. Yeah. So this is just running on port 3003 thousand 100 on my local. Speaker 2: Right? An easy way of doing this would be to use something like Engine and expose it. But to make sure things are secured, I use something called, Scale, which is just a private network with just my phone and laptop on it so that no 1 else can access it. So so that's how we take care of security. You'll have noticed there is no auth, etcetera. Speaker 2: That is because these are the only 2 devices that can be on this network. I think I'll wrap chat and stop for, questions and so on. But, yeah, to quickly summarize, this just lets you control your local machine through your phone. Took some pains to be securely. I'm ensure more can be done. Speaker 2: But, yeah, with this, you can run codecs and cloud, from your phone. Speaker 6: So you're saying you're establishing a connection between your phone and app. Uh-huh. So is it possible for you to transfer any files by chance, like, from your phone to Speaker 2: Yeah. I've tried. So, you know, you could do things like, you know, open up a a FTP or something like that on your local that your, phone can also connect to. So you, by default, it'll be a little difficult. Like, the current feature set doesn't support City, but with a little bit of workflows, definitely. Speaker 6: Maybe, like, images or something AI Speaker 2: Yeah. You should be able tool. Like, you know, the chat UI could itself be Engine, so tool support file transfer as well. So that way use can do that that too. Speaker 6: And second thing, like, suppose just by 1 on 1 interaction as you're saying with laptop, is it possible for you to access a particular file, like, by providing the path or something? Speaker 2: Yes. Yes. Speaker 6: So what are is there, like, some file you cannot access by this? Like, any, like, file type? Speaker 2: So so you can access anything that Claude and ODeX can access on your local. I've set it up such that they can access anything. Okay. So but it will depend on the individual user's preferences. If you decide that your agents can access everything, you can access everything through your phone. Speaker 2: Okay. If you have gated them if you've given them certain folders, then you can only access those folders. Okay. Speaker 1: Thank you. Speaker 4: Anybody else? Speaker 2: Smaller. Smaller site? Okay. Yeah. Speaker 7: Thanks for the demo. Just, AI was wondering, why not, for example, open, use OpenVPN or Scale to directly Chat to the, machine and use AI. Speaker 2: Like that as well. APIs Scale just seemed easier. I just had to, you know there was a GUI. I just connected both machines and it just worked. Speaker 7: Because in the phone, there is, like, the Voice juice SSH. You can just Yeah. Speaker 2: Yeah. For sure. Okay. HINTS like chat, I also have Firebase author as well, like, you know, Google auth. So those things would be the would be better to have it a little more secure. Speaker 7: Alright. Alright. Speaker 2: Yeah. Thanks. Speaker 4: Have a doubt. Anybody else? 1 final question? Then we'll take the new bus. David, if you want me to stop. Speaker 4: Yeah. My question is, you're trying to access everything on the app. Right? Mhmm. Via the mobile? Speaker 4: Yeah. So if I wanted to send out an email I know I have Gmail on my phone, but then local locally, I have Outlook also. Right? So is it possible to do chat? Like Speaker 2: Yeah. Yeah. Because if you stable, like, a Cloud or ODeX to connect to these email tools, you can tell it and it can, you know, attach a file from your local and send the email. Things like that are definitely possible. Speaker 4: So it is possible to schedule some tasks via the mobile phone? Like, let's say I wanna, post on LinkedIn or something like that. Speaker 2: Use that Speaker 4: is also possible. Right? Speaker 2: Yeah. You'll have to set up that pipeline sharing APIs, like, you know, this Courts can access your LinkedIn account and so on, but you can trigger it from your phone force sure. Speaker 4: Okay. Yeah. Got it. Got it. Ajay. Speaker 4: MCP APIs my behalf. 1 final question before we call Day. Speaker 0: AI, this private network which we reducing, which 1 is chat? How we can configure that this private Speaker 2: it it's called TelScale. This is AI a free software. You can just download it. Yeah. So you see, you know, if I go to network devices and my devices so I have this, so this app, l s l j I s, whatever is my laptop, and I have the my iPhone configured as LLC Speaker 0: plays at the end view. Speaker 4: Yeah. Ceo. Speaker 2: Okay. Zustand I'll also quickly just show it to you on my phone as well. So if I Yeah. Speaker 1: Yeah. So just Zustand Speaker 0: this private network can be connected to multiple devices also like the VoIP. Speaker 2: Free tier gives you a 100 devices. Speaker 0: Okay. Speaker 2: So you you can extend it until Engine then then it's paid. Speaker 0: And and the app which reducing, that can be connected to any of our laptop. Let's say use have 2 laptop and you want to connect AI Speaker 2: Yes. Yes. So this I have this server running locally on my laptop. Okay. So as long as you have that server on your laptop, it will automatically give you a URL that you can just access. Speaker 2: K. So here they're coming back to Zoom. You'll see that the VPN oh, sorry. The VPN is on. So this is I have the Scale app on my phone, and I have the Tailscale app on my laptop. Speaker 2: So you just have to get Tailscale on every device you want in your network, and then only they can access that network. Thanks. Okay. Speaker 1: I mean, I'll do a wrap up. Speaker 0: No. No. Speaker 4: Yeah. So so so so so. Thank you. It out. You API people. Speaker 4: Okay. Yeah. I'll get my laptop. AI.