@howo:All right, so on my end, I am still working on communities. I spent a lot of time trying to find a bug in my code, and in the end, it turns out it was an old issue in Hivemind itself. That's why I talked to you yesterday, Dan.
Long story short, subcomments are not being noted as being part of a community, so the community ID part is null, even though it shouldn't. So I'm working on a fix for that.
@blocktradesWhat is the impact? Do moderators still able to moderate comments and stuff like that?
@howoThe impact is kind of negligible. To be honest, I haven't tried any of the community features on top of it, and I know that the APIs, when you query a post, they tend to write everything under it as part of the community. So that's too much of an issue, but it's an issue for data integrity, because you have stuff that's marked as not being part of a community, even though it should. So I assume some people like me could get confused by it. And also, I'm not fully sure, because I haven't checked, but I would assume that some functions
may not work.
@blocktradesYeah, I guess, I mean, the other thing we have to be sure of is such a change doesn't somehow break something too, right? Maybe something's expected only posts to be marked this way, so it could be something that breaks as a result.
@howoTo be a bit more technical, the way we know if a post or comment is part of a community is currently by checking the parent permlink, which is like the first tag. And so it would be like hive-1111-whatever, the issue is for a post the parent permlink is correct.
For a comment, the parent permlink is also correct because the parent post would have the correct tag as well.
But then if you have a depth 2 comment, it's parent permlink is going to be its parent comment and no longer the community name
@blocktradesYeah, it's gonna look like it's, I understand.
I get the problem.
Okay, so right now, the first level comments are correctly part of it, so it seems likely that there should be no problem in marking the rest of them.
@howoYeah, exactly.
So yeah, so I stumbled upon that. That was a fun time. Now I'm deep into the PGSQL stuff because I need to fix the huge process community post stuff.
Apart from that, I had a chat with CrimsonClad, who, following up on what we said last month about the API docs, and basically, from my understanding, the API, let's say, definitions
are going to drastically change in the future, with like HAF-enabled nodes and such.
So I have started to spend a bit of time just fixing the current docs, initially, I was thinking of doing an overhaul of the whole thing.
But instead, I'm just basically going through each example, testing if it still works or if something broke, because she told me it's a source of frustrations for devs
that she's involved in.
@blocktradesI mean, I'm surprised that if anything's broken, I mean, I'm not aware of any breakage in the API.
@howoIt's not as much as the API being broken, as much as examples being incorrectly written or something is pointing to the wrong api it's really like very minor stuff for us, but pain for someone who's not used to the API.
@blocktradesOkay, so as far as what we've been working on lately, We're basically getting ready to be able to deploy a new version of Hive and HAF.
This won't be a hard fork change, obviously. But it is substantial change in many ways. And as part of that, one of the things I think I mentioned before is we're really trying to ease the whole deployment of the whole system, especially since we expect everybody to be running HAF going forward and not running the old or account history or the old Hivemind.
And to make that all as smooth as possible, We're basically setting up dockers that use a Docker compose type command to launch everything. And you can obviously edit to not launch things you don't want to launch, but in general, for a full node, we expect them to run with all the dockers that we've got specified. And so as part of that, to make this whole deployment as easy as possible, I kind of feel that we have to go to ZFS.
So we're basically a recommended way to run an API node going forward is going to be to run the API nodes, dockers and everything bound to ZFS attached storage.
This will solve a lot of problems we have with providing backups of the databases, for instance, because we'll just be able to provide ZFS snapshots instead. And that's just so much quicker and has a lot of benefits in terms of ease of updating.
Another advantage of all of this standardizing on using ZFS is we can also give a standardized layout for how to set up the Postgres database for optimal HAF usage. For instance, one of the things we found is that a lot of people will probably want to
run compressed storage for their database itself.
But if they do, they don't want to compress the key tables like the wall where the right ahead storage is for Postgres and probably even for the PG tables themselves, the sort of standard PG tables.
So we'll have a standard basically layout where you could that says, here's the uncompressed where the uncompressed data gets stored, here's where everything else is stored in a compressed form.
And similarly, we could have store your blockchain here and uncompressed storage because you're already compressing it at the high D level, so there's no point in trying to recompress
it again. So I think just saying the standard requirement is use ZFS if you want to use our stuff, doesn't prevent somebody from still using their own file storage system if they want to do it
themselves. But the stuff that we provide to make it easy is going to assume that ZFS is the way to do it.
So I definitely want to mention that. I think we're kind of, we're basically doing testing now in our own environment. We haven't quite got to production testing yet, but I expect we will in this coming week. And so I hope within the next two weeks to have a release out, assuming no issues are encountered along the way.
So that's kind of where we're, as far as the schedule goes, what we're looking at. I still, I can't promise in the two weeks for sure. There's certainly things could come up, but I feel relatively competent, but you can make it in two weeks. As far as what's going on beneath that, there's just a ton of stuff we're doing. I don't, I can't even encapsulate it really quickly in this call. So I'm not sure I even want to, unless # bartek, I don't know if there's anything in particular you think we should mention that I haven't talked about, but otherwise I'm probably just going to make a post with details about stuff we're doing.
I don't know for sure, release plans are our biggest, biggest plans and biggest efforts is we are mostly focused on that. For the other side, I can announce our new tool, Clive, which probably is made up at some quality level, able to be tried for some, for some external people. And other interesting things we are processing, but actually at some initial stages are some,
some support for other languages integrations being done by ProtoBuff integrations and definitions of high-operations written in this format. As far as we have done research for that, we were able to generate very useful steps for TypeScript, JavaScript and Python and this way actually we'll be able to offer some standardized definitions for such environments and in the future eliminate some logical copies of such operations from external libraries, like no w, like hiveJs, and I don't know what in Python probably being used.
Definitely this step will be useful, also we would like to make some much standard virtual operations, which we plan further years and probably for other depths it will be much more useful if some important comments will appear also while developing the code, because
as far as we checked the IDE support for them is quite good and you will be able just to create transactions by operating on classes and object programming. So it looks like very promising and really much better than current provided ways. Okay, that sounds great.
@blocktradesActually what you're talking about, just maybe think of something I mentioned before, I don't know if I'd ask you about, have we yet assigned someone to create an example application? I think we were talking about one, like to replace the old Lies application that would,
as an example of a HAF-app that generated transactions, did we ever?
Yes, yes, yes, and that's one of our plans and actually all of such work was involved by needs of our new HV out component and their needs to operate somehow on transactions, operations and signing them. And definitely such stuff will simplify creation of such example applications, both for Python like also JavaScript and integrations in the blockchain on such platforms shall be much simpler than currently.
@blocktradesOkay, so okay, I guess my question was if we had assigned someone to start looking into doing that app. I know that they're going to need to use the stuff you're talking about, but I thought they might be able to just start some parts of it even without the actual, I just don't know if we have anybody free there for that matter. Anyways, we can talk about that offline.
Okay, so I guess that's, like I said, I don't want to get into a lot of the details of what we're doing. I'll write more of it in the post because we could spend the whole time here talking about that if we did.
So I guess we'll just open up and see if there's any other things for discussion or if anybody wants to talk about anything they're working on.
Actually, I have a whole question to you related to the work you have mentioned. Can you check or just answer if are you sure that verifying this comments, relation to the community's will not involve any performance penalties because from your description, I suppose that you need to traverse some hierarchies of points to verify that they are related to some communities. So maybe I'm wrong.
@blocktradesYeah, I guess the question is there is there explicit marker that says they're part of the community now and or you need to add that or are you just changing the code to walk up the tree?
@howoYes, there will be performance impacts for sure because there is an extra join to be done there. The question is how much and to that I don't have the answer.
@blocktradesWell, I guess the question is how you're doing it.Are you doing it by walking up the tree or are you doing it by are they somehow marked to say this is part of this community?
@howoOh, yes there's a field for storing it, it's done during indexing by going up one in the tree.
@blocktradesYeah, you don't have to walk up the tree every time it's just on insert. you do it during indexing you walk up the tree but then after that it's done, I don't see that as a big problem if you're just doing it during indexing it's a bit annoying but I don't expect the performance hit to be this big
@howoit's mostly tricky because the database is very fragmented to be efficient and so I have to start from like a parent permlink and then join the the permlink data and then join the posts and finally get the community ID but yeah I don't expect it's to impact the performance too much and I mean it's only gonna be during indexing not during reading right okay it's not just such a big deal.
@blocktradesyeah okay that sounds reasonable to me is anybody else have anything else they want to talk about today or is it a quick day today ?
@borislavzlatanovwell I guess I just mentioned especially maybe for anyone listening if you're developing any apps on hype if you'd like to review the post in the hype that's community called standalone app apps request for feedback just see what feedback we get from the community about whether they would enjoy powering having their app powered by these two or both API nodes that'd be great
@blocktradesso I have been thinking about that a little bit just so you're aware I've been I've been thinking about how we can do it in an efficient way I think we can do it pretty efficiently do the publish subscribe if one of the things I was thinking about we could potentially do is just write the have it write the most current blocks to a table so basically have a table which just stores information about the current block and then every every every database sub database like the ones you're talking about would basically just subscribe to that one table they would just get the data about that one block push to them as soon as they get it and then they could basically do whatever they want with that data # bartek suggested another way to do it rather than pushing it through the publish subscribe would just be to provide an API for it but I think there is one advantage to the the publish subscribe method which is the data gets pushed right away instead of being pulled so we get the data more more quickly another issue that that comes to mind though is if we do this I still think it'd be nice if we could do it in such a way that basically something that looked like a traditional HAF app could operate off this data and right now that definitely would not work so I want to see if there's some way we can think about modifying or what we could do to make something that look like a HAF app even though it doesn't have all the HAF at tables run in this kind of architecture I think it's maybe possible there's definitely going to be some limitations because obviously an app like this won't have all the prior data that a true full HAF app has it's only going to have the current block data you know as it comes in it won't be able to go look in the previous history but still maybe for a lot of apps that HAF apps that might be enough I think for a lot of them it could be so but it's going to require some more thinking I think I need to I need to think about it for a couple days to see if we can reasonably make it so that we could basically take the skillet and architecture of a HAF app and run on this kind of database that doesn't have all the data.
So I'll definitely add some more to it once I've had a chance to think about it more and look over the architecture that we've got for HAF app and see if it could be mapped to it so that's about all
I had thoughts on that right now.
the proposal talks about the so-called relay app which I guess could what we were thinking is that it could subscribe to to a full note to get the stream to stream irreversible blocks right from full note without relay app and then the relay app could...
Well I don't think you need a relay app in this I mean I don't see any need for a relay app or say I think you can just have a database that's directly subscribes to HAF database I don't see any need for a relay app at first glance I think that's just writing code for that we already have functionality for unless you're thinking of something doing something that
@borislavzlatanovI'm not well it would just be I guess an extremely small like JavaScript with a postgres database
@blocktradesyeah but what I mean if you can have a database directly get the data it's going to be more efficient than anything we write that pulls the data
@borislavzlatanovyeah that's that's a good point yeah I guess I was thinking a higher number of relay apps would be advantageous for you know a more robust network so
@blocktradesyeah I mean I understand what you're saying you you want to have the idea that we could have like a hundred apps a hundred databases that anybody individually runs that only has the data that need it by some subset of the full HAF database yeah I mean I don't think it's I think the idea is reasonable and like I said it's something I've been actually thinking about too I the issues that I see really going to revolve around having an app that doesn't have as much data as a full HAF database and making sure that we can still it's we don't have to change a lot the sort of flow of control of the app because of that probably.
I think we can make that work but it needs some looking into I mean obviously another solution is we just have these have these databases running something that aren't even HAF apps at all and that obviously is pretty trivial to do but it's not as it would mean that those apps are more custom in the sense that they wouldn't be portable and things like that which I don't think is really desirable so yeah but anyways it's the idea itself and how to do it I think I think we can make something that works in some way that so that we can make light databases that can serve up API calls I don't think this isn't I don't think this is anything like we have an immediate need for because I think I think the existing HAF API HAF nodes will easily be to serve all the near term traffic but at the same time yeah we got to think long term anyways and it might be interesting for somebody who just wants to run a very small database yeah and if we want to build a bunch of apps you know I hope we would like to do that so that's a plan yeah yeah the only thing I want to do is make sure that it's we the apps get written in such a way that they're have a common architecture yes otherwise everybody has to go relearn each one individually and that just gets to be a nightmare especially if they have a lot of a lot of apps now defeat the whole purpose.
