Unfortunately, "challenge" questions are generally off-topic at Stack Exchange; to my knowledge, only Puzzling and Programming Puzzles & Code Golf definitively accept them. Indeed, the Don't Ask help page says:
You should only ask practical, answerable questions based on actual problems that you face.
which, while debatably inapplicable to many sites (Puzzling included), is still the basic rule of thumb in many Stacks — their objective is to help people find information or solutions for needs they're actually experiencing, not to have people invent problems to challenge their ingenuity and problem-solving abilities. That's likely why you were told that challenge questions are off-topic for Math.SE.
Having said that, a brief stroll through their Meta is confusing. This post suggests challenge problems are ok but best done carefully, and not all are well received, so it seems to matter how they're posed. This post asks the question directly, and the answers show the community there seems to be divided on the subject. A challenge might be seen as a mini-competition; small competitions are apparently allowed on MSE and even well received, but then there's this post saying somewhat the opposite. My guess is Mathematics.SE isn't quite sure themselves if puzzle questions are allowed.
Here, of course, puzzles are all the rage, and challenge questions are the norm (though this was not even always the case here!). But you've found that we want the focus of the puzzle to be the puzzle, and that means more than presenting a challenge question to be solved. The first of the "relevant posts" you referenced includes the very relevant statement:
There's strong consensus that straightforward turn-the-handle mathematics questions don't belong here.
Why is this? Because a "puzzle" that is just a dressed-up math problem and nothing more will leave the solver with the experience described by this answer to a related question:
The solution is standard and boring. [...] It's something you'd see in a math textbook. It's not easy, but it doesn't require any insight or creativity. You just chug ahead with the usual technique. It's not much fun and it feels like work. [...] I think it's important to avoid math problems being posted as puzzles. I feel like I wasted my time on this challenge looking for something nice that wasn't there. This weakens my trust in other puzzles that I might consider solving.
That's not the kind of experience we want to foster here - this community enjoys puzzles because they're fun, and the enjoyment is more in the journey than the destination.
Your questions sparked a bit of discussion in our site's main chat room; you can read it in its entirety if you like, starting more or less here with someone asking the same question you are asking us - why is that posting not a puzzle?
Relevant snippets from that chat room discussion:
Math is not off-topic. But math with no "aha moment" is.
100% math can still have aha moments
There are definitely "aha moments" in math. (I'd argue that a lot of the solutions to [...] puzzles are mathematics, even if the solvers don't know it.) The key isn't being "not math" - the key is being "not routine calculations".
"What's the solution to 3x^2 + 5x - 12 = 0 ?" isn't a puzzle, even though the answer is "apply quadratic formula"
[...] Something straightforward that has a predefined application (e.g. using the quadratic formula to solve a two-power equation) is hardly inspired and accepting of creative solutions.
I think a creative element would manifest itself in "I would never have expected that math construct or equation would be part of the solution here!" - I just glanced at the latest one and I have a pretty good suspicion that the answer will involve nothing I wouldn't expect to see just by that quick glance at it
[This puzzle] did seem rote to me - mostly angle and side chasing. I could be wrong, but the last one was fairly straightforward
In short, it seems your problems are too much "challenge question" for M.SE, and too little "puzzle" for P.SE, leaving them stuck in limbo.
I think Math.SE's position regarding whether or not challenge questions are on topic could use some clarification, because as my exploration through various Meta postings there revealed, it doesn't seem well settled to me.
I think Puzzling.SE's position is reasonably well spelled out in the posts you've already referenced, and hopefully this answer helps apply them to your specific questions.
In any event, your posting here does not read like a complaint, but like someone who wants to apply their creativity to provide some intellectual stimulation to an appreciative audience. You mentioned that
[...] there is thinking required in order to find a path towards the solution. There are calculations involved in finding the solution, but the solution isn't the main part. The main part is finding a path to the solution by finding some clever property in the geometry.
Requiring "thinking" is problem-solving, and that's an element in puzzling but also one in solving math exercises; it's not quite enough. Nor is requiring the solver to find something which is "clever" but nevertheless directly signposted because there's literally no other viable approach to the problem.
But I do want to especially highlight this -
I want to also add that it is suggested that puzzles should be fun/enjoyable to solve, and on both forums I did actually have positive feedback from the comments. The answers to the problems were also elegant and surprising since they are related to well-known, irrational numbers.
This is exactly the mindset we want to promote, and exactly the kind of puzzle we'd love to see in the mathematics tag! I sincerely hope your first attempts here don't put you off, because I think you've got the right idea and can indeed be someone able to provide some fresh and clever mathematics-based puzzles - genuine puzzles whose solution path leads the solver not just to have to think, but to have to think differently, to apply math in unusual or unexpected ways to find a solution that is also unusual or unexpected ... in short, to spark that "a-ha!" moment, as that's exactly what makes puzzles fun and enjoyable to solve!
I hope you'll stay with us, and look forward to seeing what you do next!
