Welcoem to posts!!
in the future - u will be able to do some more stuff here,,,!! like pat catgirl- i mean um yeah... for now u can only see others's posts :c
I was thinking of streaming engineering blog dissections live. what do you folks think? what would you prefer?
50 - 23
silver play button loading ... ⚡
It took me 4 years and 185 videos to reach this milestone. The journey was tough, but it taught me to be consistent and efficient as I am the only one doing content research, preparing notes, recording the video, and even editing it.
But, to be honest, work does not feel like work when you do things that you absolutely love :) and I realize it every time I am preparing for a video.
I am on a mission to spark engineering curiosity and elevate the ecosystem by talking about things that matter. Thank you all for resonating, it means a ton! 🙏
378 - 46
In the world where you have managed databases, should you learn sharding and partitioning? ⚡
According to me, the answer is an absolute yes! you should know these concepts and more importantly how to implement them, here's why.
1. Even when the database is managed, you can get a better performance out of existing managed MySQL and Postgres by partitioning the data, helping you improve UX and reduce cost.
Read: stackoverflow.com/questions/16499004/mysql-perform…
2. Knowing how the data is laid out on the disk and nodes to which each partition is assigned, you can pick the right partitioning key ensuring you get the best performance even from the managed database without requiring to overprovision it.
Read: aws.amazon.com/blogs/database/choosing-the-right-d…
3. The concept of data partitioning is very similar to split ownership which you would require on the compute side to understand how you would do distributed computing (and even single-node multi-threaded programs) such that you maximize the hardware utilization.
Watch: http://youtu.be/2PjlaUnrAMQ
4. Knowing these are also essential when you work with Spark because even though there are managed Spark offerings, by partitioning your data well and understanding Physical Plan, you can bring down your Spark job execution time and in turn the cost.
Read about: Broadcast Nested Loop Join
5. Knowing how to partition data without downtime, and move shards across without downtime is something that will help you do no-downtime database migrations (from one DB to another).
Watch: http://youtu.be/9iAJjtvBwyI
I totally believe these are the concepts that you should know in and out because now distributed computing and split ownership patterns are becoming quite common and Sharding and Partitioning form the crux of it, but you and others may have a different opinion on this.
In case, you want to understand Sharding and Partitioning from absolute scratch and the real difference between watch http://youtu.be/wXvljefXyEo.
#AsliEngineering
25 - 3
While recruiting, I always looked for two things - ability and culture-fit ⚡
Ability is all about skills, knowledge, and relevant experience; but more importantly, the potential to learn new things easily and make a disproportionate impact.
Culture-fit is all about how well the engineer aligns with the company's values and the team's operating style and ethics. More importantly, I looked for "ease of working with" because team culture >>>>
and nobody wants to work with a genius jerk.
121 - 11
YAGNI is my favorite principle when I build and implement any system ⚡
YAGNI is an abbreviation of You Aren’t Going to Need It, and it is all about avoiding over-engineering and sticking to the current requirements.
Given that requirements change often, due to changes in prioritization or user adoption, spending time doing things that will never be used is a complete waste of time and effort. YAGNI safeguards us against this uncertainty enabling us to iterate rapidly.
One example of this is the creation of abstract classes and interfaces for every single damn thing. You chose to abstract out the DB layer on day one because just in case you might change your database in the future is the epitome of wasting engineering bandwidth.
Such abstractions add additional code to read and understand, which adds to the cognitive load to make further changes, especially during the early days when you should be prioritizing execution over extensibility.
I am not saying not to add interfaces or abstract classes. Small efficiency habits that are not time-consuming and do not make the implementation "highly complex" are okay to implement. So, before you implement just ask yourself, is this something we should be doing right now?
ps: I have taken low-level abstractions as an example, the argument holds when you unnecessarily create microservices or go for the most "scalable" database on day zero.
89 - 4
Last Sunday I did a Twitter space talking in-depth about SQL vs. NoSQL and 700+ people tuned in and we had an amazing time discussing ⚡
Instead of just talking in general terms, we spoke about how we could implement ACID in distributed databases and the challenges that come with it using the first principles.
We challenged every single belief we had and made the entire discussion fruitful and interesting. The best part about the space was that it was not a monologue, instead, it was filled with discussions and brainstorming.
⚡ This Sunday at 9:30 PM, let's discuss and brainstorm Bloom Filters From the first Principles; a few things that I have in mind are
- how they work
- tuning parameters and how to tune them
- how to grow and shrink them (with and without downtimes)
- how to make them deletable
- and their limitations
Set the reminder - x.com/i/spaces/1vAxRAQQXMvJl
⚡ I keep writing and sharing these engineering nuggets, so if you are keen on learning them, follow along.
#AsliEngineering #FromTheFirstPrinciples
81 - 5
If you want to know how payment systems are implemented, just look at the API documentation of one of the payment gateways ⚡
The available APIs and their signatures reveal a lot of details and a few terms that should catch your eye are
- idempotency key
- webhooks and rate limit
- handling failures and retries
- static IP addresses and certs
- transient and semi-transient failures
- backoff schedule to address thundering herd
Fun fact, most payments SDKs handle DNS caching on our behalf ⚡
By the way, here's how Razorpay has built and scaled its notification service
watch - http://youtu.be/DQwlmTvs6xA
I keep writing and sharing these engineering nuggets, so if you are keen on learning them, follow along.
#AsliEngineering
16 - 2
GIPHY very smartly uses CDN to serve 10 billion GIFs every day ⚡
Here's me explaining how they do it in granular detail and in the process going into some internal implementation details of a CDN.
watch - https://youtu.be/-bo7oVejgRM
I keep writing and sharing these engineering nuggets on topics that matter, so if you are keen on learning them, follow along.
no fluff; just engineering.
#AsliEngineering #SystemDesign
29 - 1
What are zero-cost abstractions and why do they matter? ⚡
When a programming language, like Rust, provides zero-cost abstraction it simply means that high-level constructs (classes, wrapper functions, iterators, type templates, etc) will not incur any runtime overhead as compared to its low-level implementation.
For example, the following code that uses the "sum" function on an iterator will not be slower than writing an explicit loop over the vector.
```
fn main() {
let numbers = vec![1, 2, 3, 4, 5];
let sum: i32 = numbers.iter().sum();
println!("Sum: {}", sum);
}
```
Despite the higher-level abstraction, the compiler and standard library implementation are designed to optimize the code, so the performance of this approach is always comparable to the low-level alternative.
So, no matter how you write your code, no matter how much of high-level abstractions you use, the execution will remain as performant as possible.
⚡ I keep writing and sharing such engineering nuggets, so if you are keen on learning them, follow along. no fluff; just engineering.
more on arpitbhayani.me/
#AsliEngineering
124 - 4
Adding a feature to a massive open-source codebase is not difficult, all you need is to apply first principles ⚡
I just published a video where I make changes to the enormous CPython codebase and add a new statement to the Python language, named "nuke".
watch - https://youtu.be/mLNNKl0oGiQ
This is the 3rd video in the CPython internal series. Following this series will help you not only learn the internals of the Python language but will also give you a taste of how to understand and make changes in massive open-source codebases ⚡
#AsliEngineering #CPythonInternals #OpenSource
18 - 0
I am a software engineer passionate about System Architecture, Language Internals, Distributed databases, and Advanced Algorithms. I am on a mission to bring out the best engineering stories from around the world. If you are doing something interesting and want me to dissect and talk about it, drop me an email at speak.with.arpit@gmail.com.
Before co-founding Profile.fyi, I was a Staff Engineer at Google leading the Dataproc India team in providing a managed big data ecosystem to GCP customers. I hold 10+ years of experience in building and scaling backend services across domains and companies like Unacademy, Amazon, Practo, and D. E. Shaw.
I keep diving deep into engineering details and share my learnings by across my socials and videos on YouTube. To put my learning to practice, I keep building things on the side and a few of my good hobby projects include
- DiceDB - a re-implementation of Redis in Go
- Revine - a visual programming language for kids