Data Entry automation using Browser Agents

Speaker 0: MoE at the talk, lead and clear. Okay. So thank you, Scene, for, you know, this event run inviting me here. A very quick you know, Jun wanna add up to what he said because there's a lot of people new to this AI Tinkerers event. I think 1 of the best events that I've been part of File, I'm a part of a lot of AI groups. Speaker 0: The reason why I like to come to this mainly is because it's always consistent. A lot of a lot of times, like, you know, author, let's say, I do a MoE, we're done with the Demo, then we have the the networking part. The the discussion that we end up having is is the 1 that makes me think of what are the next demos I can work on. Sometimes, you guys Jun run normal discussions or questions, I get, like, ideas what I can work on for the next project or enhance what I actually demoed. And in fact, the demo that I'm gonna be showing today is actually a AI Inside we had at work run we wanted a solution as soon as possible. Speaker 0: And the only reason I knew, like, this specific framework is because in 1 of the very 1st sessions I had, it it AI, someone told me, like, is there an AI agent that can do x AI zed? I looked it up. I found this. So very quick introduction about me. My name is Yazan. Speaker 0: I'm actually a biomedical engineer, but then I've transitioned to the AI. So my whole Live, at work, I'm more of an AI engineer that works on AI and health care solutions. And just before I get started because there's a lot of new faces here, I wanna know your AI' background because that gives me and the other speakers an idea how we can improvise AI we're presenting our solution. So, like, I wanna know Jun, like, are you guys Pandini didn't wanna do it manually run we had AI I think 1 or 2 days just to figure something out. And thinking this this is like the solution I end up I end up coming up with and I'll be demoing. Speaker 0: So I'm not gonna focus a lot of I'm not gonna focus a lot on the, part AI reading from Excel AI Founder so on. My focus would be on the agent talk act the agent tech part. So just to give you a very quick AI. So an input file would look something like this. So we had AI to the prevent we worked on the, talk, the DAGs, and then, how many hours worked on each 1 each day. Speaker 0: So this is how the input file we lead, which is what we wanted to put in our new system. But again, there was no API integration or we had to pay a lot of money. So I just want to give you an idea of how this actually looks AI. And what it will end up filling is basically this. So this is the new system we're migrating to. Speaker 0: So it just has Artist of prevent, Day, time, the hours Use worked on each task. So that's the whole point. This is an actual use case. We had to come up with something that would automate doing the data entry for all of the Excel AI we showed you. So the Excel file is just like a couple of rows that I'm gonna show for the Demo. Speaker 0: But in reality we like MoE had 1,005 entries to do run no one's gonna do that manually. So 1st I'm going to show AI going to like go focus on the main part of the demo of the code handle then I'll run it so you guys can have an idea how it actually works. So I'm gonna skip this part where again APIS part isn't the main 1 where it's just like for example APIS class is just printing the logs and writing the terminal. I don't wanna focus on that a lot. So this is basically TTS AI schema. Speaker 0: So we have Date tag hours which is AI you've seen in the excel file run the whole point is that imagine you have 10,000 draws and somewhere in the middle something Browser, it failed, it missed whatever. You want to track actually what missed so that once it's done with the separation, you can go back and fill those that it failed manually. So that's the whole point here. Now a very quick idea about what this agent BITS. It's what we call browser agent. Speaker 0: So basically you would prompt it and let me just zoom in a bit. Speaker 1: Guys Speaker 0: at the talk, can you see the screen or do I zoom in a bit more? Lead. APIS is better or AI all good? Okay. So this is a browser agent and what a browser agent is that you would prompt it. Speaker 0: Let's say go to a specific URL and it would actually go to that URL and do actions for you. So imagine you have an agent that's controlling your computer that would do talk on your behalf while you're working on whatever main Fast you have to do. So 1 of the problems I have is called speed optimization problem because it is actually quite slow. But again, we do not we do not care about the AI. We Jun cared about something that would actually do it for us. Speaker 0: And that's why here this is an attendees extension of the system message which I call it speed optimization prompt. So if you go here, you'd see like BITS extremely concise and get to the goal as quickly as reusable. Assume the pages are ready and so on. And then here there's something called chain of draft. I'm not sure if many of you guys are familiar with chain of draft. Speaker 0: You MoE probably heard of chain of thought, right? Chain of thought is basically when you give it lead say Day step by step down how to do a certain task. Chain of draft is something similar BITS then for each step you're telling it think in maximum 5 words. This is actually backed by across research paper which came out 2025. It ends up using about 97% less tokens with the same performance as chain of thought. Speaker 0: So I tried to implement the chain of draft here to like thinking, just like up to 5 words. So if you see here, limit each visit to around 5 words and so on. When I tried with the browser agent, it really didn't do it properly, because simply because it had its own built in system instructions. So that's why it was kind of, AI, getting confused. Like, do I think in 5 words, or do I use the instruction that's, you know, from the open source, framework? Speaker 0: And here, again, this this part is Jun, like, accessing the Excel file, passing in the keys, and so on. So I'm skipping here. This would be on GitHub. So if you guys wanted to get go back and check, you can ask, let's say, g p 2 or plot, explain what this does. You can do that. Speaker 0: For example, this is something 2026 I would talk about. Let me remove this run now. It said I left MoE, CSK? It shows that I left Zoom. Pick me out of Zoom. Speaker 1: Right. True. Are You AI by any handle. It kicked me off. Your AI Fi got disconnected. Speaker 1: Just try again. AI City disconnects after a few minutes. Just a reminder, if you guys wanna connect to Wi Fi, it's BITS RESTful those who are not BITS, like, from bits. Keep in mind that it disconnects every half an hour or MoE, so just don't be surprised. We'll be back in just a minute. Speaker 1: So go to your email also and approve it so that you get, like, a few more TTS minutes extra. Yeah. So Jun, yeah, just click that and do it. So Is that Training? Jun. Speaker 1: I think it's the peak safety. Jun zoom in again. Just refresh run. AI. Because I think it would Post Scene it now that you have Wi Fi coming in. Speaker 0: Oh, wait. I'm connected now. Speaker 1: Yeah. Yeah. But it'll Post gonna have AI minutes of run Date, AppliedAI, if it doesn't, what's that File? Verify you. Check out the emails. Speaker 1: Right? AI, it just Date. Lead would have layer. Yeah. That's run. Speaker 1: Yep. Just verify that so you Use a you get about 15 minutes extra. Speaker 0: That's alright? Speaker 1: Yeah. Because Zoom MoE, also, it just just kicked me out. Speaker 0: Send here. Speaker 1: Access network. And now it'll be fine. Just open up Speaker 0: Zoom. Speaker 1: Zoom. Yeah. Rejoin. I'll I'll Yeah. Go ahead. Speaker 1: Just Speaker 0: if I Speaker 1: just do mic test app. Just do mic test. Oh, can you hear me? Yeah. Go ahead. Speaker 1: Yeah. Yeah. Go ahead. Speaker 0: Yep. Use. Sorry for the technical issues. Yeah. Just to quickly recap, this is a browser agent that basically will go to 2026 browser or link that you give it to handle start doing actions. Speaker 0: So as I said, I'm skipping the parts where it just handles things in, like, terminal logs and so on. But then let's say if we look into this function, 1 of the things what I'm doing is so if I go back to the Excel file here, what I'm actually passing is for example, when I give it the day and date, I'm also passing all these values here. So I'm passing it as a Post, and, it it's it needs to, like, fill for all of them 1 by 1. Now the issue is what if, let's say, 1 of them File? Or, let's Day, it it did the 1st 2 handle then it skipped on the 3rd 1. Speaker 0: And that's the whole point that you wanna track what is the status of each 1 so that if 1 File, you're not gonna miss that. So that's the whole point of the reconcile AI. Is that what I'm trying to do is I'm trying to track them, for each of the, current requirements that LLM, asked it to do. Yeah. So this is the main Post. Speaker 0: Now here, you can actually use any model as long as it supports vision MoE you can use Claude, GPT handle so on. From my testing Claude Sonnet to be specific is the 1 that actually does the best and it doesn't really hallucinate But I haven't tested much with GPT because, like, I just wanted to I just wanted to get this done as soon as possible. As I said, this is an issue we have at work, and we lead, like, 1 to 2 days to figure it out. And I know, like, Sonnet usually, when it comes to browser agent, Sonnet is known to be the best. And, actually, the framework, which is called browser use, they have fine tuned claw Claude Sonnet for their specific, like, framework itself as well. Speaker 0: So so so that's why here I AI Claude Jun and then Live you OpenAI AI Date extraction model is so what happens is that there's shared another model that you can use to extract the text within the page that you Fast TTS to go to to pass that as Context. So if you're only extracting text you don't want to use a big model or an expensive model to do that which is why here I'm using Haiku in this Date. So it's a much smaller model and it's much cheaper. Now the browser profile here, you tell it how long you Shaun it to wait to load for a specific page. You know how sometimes it takes like 5 seconds, 1 2nd, 3 2nd. Speaker 0: This is where you play around with the AI. And then headless is basically if you want to see the the things BITS it's actually working or not. So now I've I've kept it false so that when I run it, we Ngom we Canvas see how it actually does the actions, but then you can keep it run and it will actually execute in the background without you Institute you without you knowing. Now, here I'm specified only for the allowed domains. What actually happened when I was you know June my building phase, it ended up going 1 specific URL that led to another URL and so on and it got confused. Speaker 0: It shared getting confused. I can't do this action. I'm going to end this, this APIS run. So when I when I LiveKit the allowed domains, even if by chance it Jun an external link or even if the URL that you had has other links that, you know, talk clicked by mistake, it build redirect it back to what you told it. So which is why here Live the allowed domains. Speaker 0: Keep alive Training basically, I'll show you in the MoE, but it would require me to log in and Entry AI the authentication code. So do I have to do that for each and every single entry? No. You Shaun just do it once and let it talk constantly run for all the, project or the AI entries I'm working on. I'm skipping the Excel part. Speaker 0: Yeah. So this is basically the main so this is the Arabic automation prompt. So here, this this is how you normally prompt a browser agent. So Fast, I'll tell it go to this link. If you run not already logged in, log in with the details provided then click submit. Speaker 0: If you're asked for IDENTIFICATION automation, which it does actually, then wait just then just wait 5 seconds as it will be written automatically teaching the human in the loop part which I will do it basically. The reason why I had to prompt that is sometimes it will be like 1 2 seconds and you're not done writing the code. It Jun, like, tries to enter, refreshes the Date, so it AI messes up. When you tell it to wait 5 seconds, wait 10 seconds, it actually ends up waiting. And then if already logged in, skip Ngom entirely. Speaker 0: So if we go back to the 2nd step, in the middle of the page, you will see a date navigation control in the format Ady d m m y y. Change the date to whatever data I'm extracting from the Excel file. Also in that format, mentioning that specific format actually change a lot because it used to get confused sometimes. So it could do, let's say, 10 entries AI. On the 11th, it gets confirmed. Speaker 0: Like, oh, I did not know it was in that format. So mentioning the format of the browser along with how you're passing it from the Excel file just 2026 me skip an entire step. If you're already on that date, ignore. Now just to show you what I meant by the date navigation control here. Speaker 1: It's Speaker 0: this 1. It's basically this. Guiding it so just like how you'd normally show an LLM, like, when you use a vision model, just like how you typically tell it where to look at, this actually use a vision as well. So that's why if you tell it to look here, it's on the top left, top right. It actually understands what you say what you mean. Speaker 0: And then the remaining are the same. Go to this. Click here. Click that. So I'm just gonna run it just, you know, due to shortage of time. Speaker 0: So what it will do right now is it will open a separate browser to the link directly. And you guys might ask, okay. Am I really gonna pass my credentials to an l m, you know, security reasons and so on? In the browser itself or in the framework, there's actually a part where you can pass sensitive data. So in my case, I put the email password there Jun as you can see, it did fill it automatically. Speaker 0: Now it will click on Ngom. So if you remember the prompt I was telling it to 2026 in. Speaker 1: So that k. Speaker 0: Now here this is like the human Zoom part which I'll do it quickly. And this is the part where I said where I'll keep it alive just so that I don't have to repeat this process AI TTS and every single time. So I'll just show you how it looks shared it blocks. So it'll basically show some boxes. And then within the terminal, if we look at the terminal let me Speaker 1: just Speaker 0: expand this. You could see the the steps it's taking navigating to the URL, wait 5 seconds, resizing for screenshots. So it does actually take screenshots so AI can know what it is, but you can turn it off if you want. It says clicked here, resizing screenshot again. You can limit as well the number of steps you wanted to do. Speaker 0: So there are actions handle there are steps for a browser agent. You can have multiple actions within specific step. You can even play around with, like, how many actions per step, how many steps for the entire process, and so on. So if you look here, it says Day is now 11. Page shows the week of 08/06, but showing is very visible. Speaker 0: And then showing TTS Day picker, it shows 11 '5, but columns still show whatever. So it's reasoning with itself to know what the next steps City should do Jun then it clicks. So if we now quickly go back here, you know, now we can see that it changed it in this case. So it took it went to the correct Day. And if we just go back quickly to the terminal yeah. Speaker 0: So now it's showing to product. It says projects are visible. Where is it? Yes. Process discovery, blue printing, workflow design build. Speaker 0: Need to click on need to click on Monday, set for process discovery. So you even you can even tell it what to click, how to click, if you want it to scroll or not. So this actually takes quite some time Pandini can you Jun Date to shortage of time, I'll have to, like, I AI, end it right here BITS, this will continue running until it it finishes the entire task. So if you have 1,000 rows, 10,000, 100,000, as long as 2026 keep the browser alive, TTS continuously run. And DAGs AI tried it running for about 4 or 5 hours, it done I don't know. Speaker 0: I don't remember the number of entries, but it only eliminate on 2 of them. So detailed prompting actually helps. So even if you tell it click enter, it Canvas click enter. If you tell it click tab to go to the next tab, it does that. So I'll recommend if you guys wanna have a look at the workflow. Speaker 0: So I'll just show it here. It's called browser use. It is very fun to work on, and you can do a lot more than what what I showed. This demo was only 1. I've used it a couple of AI. Speaker 0: And you can actually automate a lot of the talk that you need to do without any API integration. So that's it from my side. I think we're out of time, I think. Right? CSK? Speaker 0: Yeah. So that's it from my side. I'm I'm more than happy to show, like, the remaining terminal after, like, the like, during the networking session. Handle, yeah, that's it. Thank you. Speaker 1: I'll just take 1 mic. You keep on. Any questions? I Day. MoE I have a question regarding the browser that shared using. Speaker 1: So how do you, see if we update the Model that you're using? Okay. Okay. So how do you safely Speaker 2: update the browser Jun' model itself? Because sometimes when you Day, like, for example, like, g p t 5 releases a patch or Claude releases a patch, how do you, like, safely update without breaking this entire framework? Because sometimes, you know, you might get a different answer or some you might get a, like, an answer that you don't wanna see from an agent. So I'm just curious on how do you safely update that that doesn't break your framework, basically. Speaker 0: There's 2 things you can do. So in pretty much any agentic workflow, so this is Browser, but then if you talk about Pilani Shaun and I don't know, create AI, there's always a there's always a parameter that says you can you can write your own system message completely. So on top of so in this case here, I'm just extending BITS. But then in almost every single agentic framework, you can write your own system message that will AI whatever it had default. So this is more of you how you really wanna prompt Institute a lot of, like, the way BITS prompt a browser Jun is totally different from Norman LLM or Greg or voice Agents and so on. Speaker 0: If you see here, if you remember the prompt, I was AI, Day teaching it Ngom here, Skill, right click, enter, tab, and so on. But then you would even do as much as you can within with AI Excel. Like anything you can do with Python and avoiding the element Post, it's better to do it. This way you can guarantee that there's less hallucination that can happen HINTS specific areas handle just the more the hallucinations will be more of the actual task. Therefore, if you've seen if you've seen the very 1st AI identic schema, I'm actually tracking each and every single error it's doing. Speaker 0: So if it failed the task, if it failed let let's for example, 1 of the things was that from my Excel file TTS said like 4 hours BITS then it throw the data HINTS 2 hours. It flags that talk oh BITS said 2026 4 hours or 2 hours lead me flag it for you. So this is how it is in terms of the model Excel. You can't like you can't change the model while it's working. You'd have to stop it, change the model, and then, lead it again. Speaker 0: Does that answer your question? Speaker 1: Yeah. Any other questions? Speaker 0: Enrico. Speaker 1: Yeah. Just use your thumb. Just use the 1st half of your thumb. Try to press it. Yeah. Speaker 1: Just use AI this. Sorry. Yeah. No worries. No worries. Speaker 1: Use the mic. I'll just put this out for the Zoom teaching. Speaker 3: For recording. Hello? So what is your experience of, Browser use when when you have to deal with the web sites that have, like, concerns. Speaker 1: So Speaker 0: if you've Scene Speaker 3: I saw I saw the thinking. There was the human in the middle, and you actually did the enter the code and all that. Yes. So the the is there anything to bypass that kind of, Speaker 0: Here in the Zoom Post? Speaker 3: Yes. Speaker 0: Yes. So again, 1 of the features or parameters you can play is that if you wanted to use the vision teaching if you wanted to actually talk screenshots as AI showing and look past that. So let's say how 2026 capture you know MoE like sometimes the letters are like kind of you know weird looking shape and there's like it's like alphanumeric numbers and so on. When it takes screenshots you can again guide it Live you will let it know that you're gonna have to solve a capture. It's showing let's say be in the middle of the page. Speaker 0: It's going to be, let's say, 5 characters maximum. It will take a screenshot. It will see that and then it will actually start writing or trying to input. So to try to capture, it might fail maybe instead of an MoE or 2 2026 0. It File, tries again 3 Realtime handle then you can actually or you can prompt it to let it know. Speaker 0: Day. I failed 4 or 5 times. Let me pass it to the human to do to do it himself. So you have that you can play around with with that point. Speaker 1: We have time for 1 final question. Who is this? You're TTS AI constraints? Let Jun just check with that if it works. If I'm AI, this File always the 1st shared. Speaker 1: No. It's a 2nd. It's a doc. I can just AI the issues. Use this. Speaker 1: Patches. So Yeah. I just want to ask, like, when you are implementing, do you think at the current stage of AI, Canvas reliably use this or, like, traditional scrapping is even better at this point? Speaker 0: You mean you mean if we can reliably use browser AI? Speaker 1: No. Like, is it usable? Is it, like, time saving or is it Yeah. Speaker 0: Like, the whole Post, time saving I mean, in this case, it took a lot of AI. But then in our case, we Day not care about the time. We just wanted a data entry solution. So that's why in this case, maybe it doesn't show here, but it opens another tab MoE AI can work in the background while you're focusing on your main, you know, job or whatever. Whether it can be Use AI, I thinking, yes. Speaker 0: Again, as I said from my testing, like, on my personal data when I run it, it only did 2 mistakes. 1, because it we had, like, the system that I showed you, AI, Realtime sheet, it had a different name that from the data Entry, Excel file. So that's why I got confused. The 2nd was it just actually hallucinated. So, honestly, I was surprised. Speaker 0: I thought it would do a lot worse. But, again, when it comes to Jun, run agentic framework, you can either override the system prompt of the actual framework or you can extend it. In my case, I just attendees it directly, and it worked fine. But normally so for example, if you've seen the chain of draft mechanism, I said, it didn't work because it was getting conflicted with the main system message. Whereas if I if I had, let's say, more time of testing, I would actually try to override the system message and use the chain of draft fault because chain of draft actually works perfectly fine for you. Speaker 0: So here you I'll give an example of chain of draft. So maybe here AI Skill Day, I would have to go to this date, click here, and then change to to the specific date. Chain of draft would be AI, I have to change the date to this directly. So it AI, like, couple of words, 5 words, and it can do the same with like, it does that with the same accuracy. So this is, like, 1 specific use case, but I've had it, let's say, scrape from different websites handle then passing the scrape data TTS Google Sheet and so on. Speaker 0: It that efficiently fine. So Live any other application, the prompting is extremely important, especially especially when it comes to, like, browser agent or voice Agents. And then 2nd, the type of model you pick. Any other questions? Or Anything else we need to Okay. Speaker 1: Thank you, Speaker 0: Azan. Thank you. Thanks.