Anonymous 1123
Let be sphere with radius R and a cone  inscribed the sphere with high is h, radius r = sqrt(2*h*R - h*h).

![ScreenHunter 799.png](/image?hash=10a8c932cbe70acf677e4216eef9b3f7dfba19b8e16fe918283460772e9b105e)
![ScreenHunter 800.png](/image?hash=3d0dddb23727fae57dd430a0ff79c3cd659fc4ce7042a243a7b888390dd4474e)

I tried

\documentclass[border=2mm,tikz]{standalone}
\usepackage{tikz-3dplot}
\usetikzlibrary{calc,backgrounds}
\usepackage{pgfplots}
\begin{document}
%polar coordinates of visibility
\pgfmathsetmacro\th{65}
\pgfmathsetmacro\az{110}
\tdplotsetmaincoords{\th}{\az}
%parameters of the cone
\pgfmathsetmacro\R{(2 * sqrt(2) * \r) / 3} %radius of base
\pgfmathsetmacro\v{(4 * \r) / 3} %hight of cone
\begin{tikzpicture} [scale=1, tdplot_main_coords, axis/.style={blue,thick}]
\path
coordinate (O) at (0,0,0)
coordinate (I) at (0,0,{sqrt(\r*\r - \R*\R)})
coordinate (B) at (0,\R,0)
coordinate (A) at (60:\R)
coordinate (S) at (0,0,\v)
coordinate (C) at  ($(O)-(A)$)
coordinate (M) at ($(A)!.5!(B)$)
coordinate (H) at ($(M)!1/2!(S)$)
;

\foreach \v/\position in { O/left,A/below,S/right,I/left} {\draw[draw =black, fill=black] (\v) circle (1pt) node [\position=0.2mm] {$\v$};
}

\draw[dashed] (S)--(O)  --(A)-- cycle (I) -- (A);
\pgfmathsetmacro\cott{{cot(\th)}}
\pgfmathsetmacro\fraction{\R*\cott/\v}

\pgfmathsetmacro\fraction{\fraction<1 ? \fraction : 1}

\pgfmathsetmacro\angle{{acos(\fraction)}}

% % angles for transformed lines
\pgfmathsetmacro\PhiOne{180+(\az-90)+\angle}
\pgfmathsetmacro\PhiTwo{180+(\az-90)-\angle}

% % coordinates for transformed surface lines
\pgfmathsetmacro\sinPhiOne{{sin(\PhiOne)}}
\pgfmathsetmacro\cosPhiOne{{cos(\PhiOne)}}
\pgfmathsetmacro\sinPhiTwo{{sin(\PhiTwo)}}
\pgfmathsetmacro\cosPhiTwo{{cos(\PhiTwo)}}

% % angles for original surface lines
\pgfmathsetmacro\sinazp{{sin(\az-90)}}
\pgfmathsetmacro\cosazp{{cos(\az-90)}}
\pgfmathsetmacro\sinazm{{sin(90-\az)}}
\pgfmathsetmacro\cosazm{{cos(90-\az)}}

\draw[dashed] (0,0,\v) -- (\R*\cosPhiOne,\R*\sinPhiOne,0);
\draw[dashed] (0,0,\v) -- (\R*\cosPhiTwo,\R*\sinPhiTwo,0);

\begin{scope}[canvas is xy plane at z=0]
\draw[dashed] (\tdplotmainphi:\R) arc(\tdplotmainphi:\tdplotmainphi+180:\R);

\draw[thick] (\tdplotmainphi:\R) coordinate(BR) arc(\tdplotmainphi:\tdplotmainphi-180:\R) coordinate(BL);
\end{scope}

\begin{scope}[tdplot_screen_coords, on background layer]
\draw[thick] (I) circle (\r);
%\fill[ball color=orange,opacity=1] (T) circle (\myr);
\end{scope}
\end{tikzpicture}
\end{document}


My code does not true every cases. How can I get a general case?


user 3.14159
I hope that I interpret the question correctly. In the plane that contains the base of the cone, there are two critical angles.
1. \alphacrit is the angle at which the boundary lines that connect the base with the tip attach to the base circle.
2. \betacrit is the angle of visibility of the base circle because the rest gets covered by the sphere.

As far as I could see, you only compute \alphacrit. I also used the opportunity to shorten the code a bit. For instance, tikz-3dplot stores the view angles in \tdplotmainphi and \tdplotmaintheta, so we do not need to introduce additional macros. I also stored the parameters in functions, which I named according to the description at the beginning of your question.


\documentclass[border=3mm,tikz,fleqn]{standalone}
\usepackage{tikz-3dplot}
\usetikzlibrary{backgrounds}
\begin{document}
\tdplotsetmaincoords{65}{110}
\begin{tikzpicture}[tdplot_main_coords, axis/.style={blue,thick},
declare function={%R=3;h=5;r=sqrt(2*h*R-h*h);
R=4;r=R*0.8;h=R+sqrt(R*R-r*r);}]
\path (0,0,0) coordinate (O)
(0,0,h-R) coordinate (I)
({r*cos(60)},{r*sin(60)},0) coordinate (A)
(0,0,h) coordinate (S)
;
\foreach \v/\position in { O/left,A/below,S/right,I/left} {\draw[draw =black, fill=black] (\v) circle (1pt) node [\position=0.2mm] {$\v$};
}
\draw[dashed] (S)--(O)  --(A)-- cycle (I) -- (A);
%
\pgfmathsetmacro{\alphacrit}{acos(min(1,max(r*cot(\tdplotmaintheta)/h,-1)))}
\pgfmathsetmacro{\rdisc}{(R-h)/r*cot(\tdplotmaintheta)}
\pgfmathtruncatemacro{\itest}{\rdisc<-1?2:(\rdisc>1?1:0)}
\pgfmathsetmacro{\betacrit}{asin(min(1,max(\rdisc,-1)))}
\typeout{\alphacrit,\betacrit,\itest}
\begin{scope}[canvas is xy plane at z=0]
\draw[dashed] (90+\tdplotmainphi-\alphacrit:r) -- (S)
-- (90+\tdplotmainphi+\alphacrit:r);
\ifcase\itest
\draw[dashed]  (\tdplotmainphi+180-\betacrit:r)
arc[start angle=\tdplotmainphi+180-\betacrit,
\draw[thick]  (\tdplotmainphi+180-\betacrit:r)
arc[start angle=\tdplotmainphi+180-\betacrit,
\or
\or
\fi
\end{scope}
\begin{scope}[tdplot_screen_coords, on background layer]
\end{scope}
\path (current bounding box.north)
node[draw,above=1em,text width={2.2*R*1cm}]{%
In order to determine $\beta_\mathrm{crit}$, we project a latitude circle
$\gamma_\mathrm{lat}(\beta)=\bigl(r\,\cos(\beta),r\,sin(\beta),v\bigr)$
on the normal to the screen,
$n_\mathrm{screen} = \bigl(\sin(\theta)\,\sin(\phi), -\sin(\theta)\,\cos(\phi), \cos(\theta)\bigr)\;.$
We can absorb $\phi$ in a redefinition of $\beta$. The solutions of
$n_\mathrm{screen}\cdot\gamma_\mathrm{lat}=0$ are then
$\beta=\beta_\mathrm{crit}=\arcsin\left(\frac{v \cot (\theta )}{r}\right)$
and $\beta=180-\beta_\mathrm{crit}$.};
\end{tikzpicture}
\end{document}


![Screen Shot 2020-08-17 at 8.47.50 PM.png](/image?hash=a4e6484dc58040e4f1686ddea409d137c57e657a5c07aeb23d8154c7b04ae14a)

The role of \betacrit (and the cases distinguished by \itest) is to draw the base circle only on the forefront of the sphere. The following animation is supposed to illustrate this.


\documentclass[border=3mm,tikz]{standalone}
\usepackage{tikz-3dplot}
\usetikzlibrary{backgrounds}
\begin{document}
\tdplotsetmaincoords{65}{110}
\foreach \Lat in {-85,-75,...,85,75,65,...,-75}
{\begin{tikzpicture}[tdplot_main_coords, axis/.style={blue,thick},
declare function={%R=3;h=5;r=sqrt(2*h*R-h*h);
R=4;r=R*abs(cos(\Lat));h=R+sin(\Lat)*R;}]
\path[tdplot_screen_coords]	(-R-1,h-2*R-1) rectangle (R+1,h+1);
\path (0,0,0) coordinate (O)
(0,0,h-R) coordinate (I)
({r*cos(60)},{r*sin(60)},0) coordinate (A)
(0,0,h) coordinate (S)
;
\foreach \v/\position in { O/left,A/below,S/right,I/left} {\draw[draw =black, fill=black] (\v) circle (1pt) node [\position=0.2mm] {$\v$};
}
\draw[dashed] (S)--(O)  --(A)-- cycle (I) -- (A);
\pgfmathsetmacro\alphacrit{acos(min(1,max(r*cot(\tdplotmaintheta)/h,-1)))}
\pgfmathsetmacro{\rdisc}{(R-h)/r*cot(\tdplotmaintheta)}
\pgfmathtruncatemacro{\itest}{\rdisc<-1?2:(\rdisc>1?1:0)}
\pgfmathsetmacro{\betacrit}{asin(min(1,max(\rdisc,-1)))}
\typeout{\alphacrit,\betacrit,\itest}
\begin{scope}[canvas is xy plane at z=0]
\draw[dashed] (90+\tdplotmainphi-\alphacrit:r) -- (S)
-- (90+\tdplotmainphi+\alphacrit:r);
\ifcase\itest
\draw[dashed]  (\tdplotmainphi+180-\betacrit:r)
arc[start angle=\tdplotmainphi+180-\betacrit,
\draw[thick]  (\tdplotmainphi+180-\betacrit:r)
arc[start angle=\tdplotmainphi+180-\betacrit,
\or
\or

![ani.gif](/image?hash=246b8e725d73882dd5ead506334d81a15db17b9576d51baf5895326dd9a3d4be)